Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 1 year ago

MODERATORS

communick@academy.garden

Improve developing efficiency in pySpark? [Discussion] (alien.top)

submitted 11 months ago by Davidat0r@alien.top to c/machinelearning@academy.garden

8 comments fedilink hide all child comments

Hey everyone, I’m fairly new at the field and I’m working on a regression model in a huge dataset. We use pySpark for it, since the full size is around 150.000M rows.

Given this size, every little step of the process is painfully slow. Every count operation, display, etc.

I have of course tried to sample the dataset toa fraction of the original while I work on the development of the model (like df = df.sample(0.00001)) but it doesn’t really make much of an impact in time. I tried sampling it so that the reduced dataset would only be 1000 rows and a display operation still took 8 minutes to complete.

I have tried to filter the data as much as I can but the smallest I get is around 90k million rows, which is still pretty damn gigantic.

I also tried to save the “smaller”, filtered dataset in disk (took 3.64 days runtime to save) and reading from that again the next day but same result: still very slow.

This is really slowing me down as (probably due to my own inexperience) I do need to do a lot of displays to see how the data is looking, or check number of rows etc. So I advance really, really slowly.

Do you, overlords of machine learning, have any tricks, tips or ideas for working with such humongous datasets? I do not have the possibility to change anything about the system configuration (btw it’s in Databricks) so I can only implement ideas via code.

Thanks in advance! David

you are viewing a single comment's thread
view the rest of the comments

[–] jacobgorm@alien.top 1 points 11 months ago

I think you're right about the lazy eval. Can you somehow materialize or dump/reimport the 1000 rows view to use for experimentation.

FWIW sampling 1000 rows at random is the same as permuting the entire dataset at random and reading out the first 1000 rows, not sure if that would be feasible or help in your case, but merge sort would make this an O(n log n) operation, so in theory it should not be too horrible.