I am going to be using aws Data Pipeline or Glue or DataBrew service to move data from one S3 bucket to another with some transformation in between. What can I do to validate the No of records between source and target? Is there any recommended AWS service?

like
Posting as :
works at
You are currently posting as works at

In aws console go to your bucket and click actions then go to get total size, then you should see number of objects in your bucket. Can also use aws cli (below)

aws s3 ls s3://bucketName/path/ --recursive --summarize | grep "Total Objects:"

like

Hope one of these helps :)

Not to completely disagree with my fellow slalomer :) but I assume the data could be in parquet or some other format where there’s compression and the size if the file won’t help you (this method would also work for CSV). You could simply create hive tables in the glue catalog overtop the s3 files (one table for source and table for the transformed data) and run a simple sql count on each table from Glue. Aka you don’t need databricks for this.

like

Related Posts

Hi guys I am posting this again I have offer of almost same package from Accenture and IBM . Accenture offering is very slight more but that's not my key consideration. Which company is better in terms of job security, work life balance. That's the top most factor I am considering. Learning and project and latest niche technology is the other important factor after first 2 factors .

like

Pros and Cons of working at Cerner?

funnylike

When is intent to apply? Ready / anxious to start the process! 😬😃

like

Designer and architects who have made the jump from consultancy/studio to brand/client side, what's your reasons and how have the transition been?

Larsen & Toubro Infotech If we join as P4, how much experience is required to get to P5 ?
Larsen & Toubro Infotech

like

What are timings in Novartis, Hyderabad? I understand it varies from projects, the geographical area of the project.
I'm specifically interested to know the timings of teams working on US based projects.

Also, is all the hype on WLB true in Novartis

like

Any ways to fake lta

likefunny

Is AVEVA good company to join as Project Manager?

like

How do you get a job working in food science research? What are some of the best companies to work for in that field?

like

Does anyone know what a chief sonographer should make? Managing 15 employees and 4 locations?

like

Any scientists or senior scientists here who work in the consumer goods industry in Toronto?
What does the payscale look like?

like
like
like

Looking for Ramadan specials to watch and bond with my family. Anybody have good suggestions? Thank you in advance

like

Entire fucking Ritz Carlton smells like cigarette smoke... how many points

like

Would you rather have 5x amex MR points or 3x chase UR points for each dollar spent on flights? Why? Company let's me book flights on personal cards and trying to decide which to choose.

like

American Airline q - if I have Gold status and an American Airlines credit card - does that mean I get two free checked bags or one

like

What cameras do you have, and if multiple, which one is your favorite?

like
like

Would anyone be willing to provide the name of a good recruiter for tax/accounting in Los Angeles or should I go to Robert Half, etc.?

like

Additional Posts in Data & Analytics Consultants

Folks applying for Data and Analytics roles in the industry at M, SM, D Levels, what technical/functional topics do you prepare for on the Analytics side?

like

Noob question - I want to consolidate the dates into a single dates column. Would numpy or pandas help me reorganize this dataset effectively or should I do it within excel?

Post Photo
like

Single book recommendation for user research?

Goal is interview prep. Ideally focused on applications in tech.

like

Quick survey: Do you guys think having a template package to envelope an ML model as an application would be helpful. The package could contain boiler plate template for unit test, docker image, CI/CD, database connectors etc and act as a starting point for doing that. Tech firms usually have that internally but limited options in open source.

like

Is there a good online course on storytelling with data? Something that can help structure your thought process and design choices when creating visualizations and Dashboards in Tableau, Power BI etc.

like

Hi! Does anyone know what kind of knowledge is needed to crack the Data Scientist interview at Google?

I'm currently working in Marketing Analytics (primarily using SQL and a bit of pyspark). Apart from mastering SQL, I wanted to know from people working at Google what does it take to get there as a Data Scientist? (I'm very interested in Strategy Ops)

I plan to another 6-8 months in preparation and upskilling myself before I even apply.
Any help and guidance is appreciated! Thanks :)

like

What tech companies are best for Data Analysts? Looking for $120k base + equity that’s gonna be worth a lot + other great benefits + possible remote.

funny

This might be a long shot ... but I have a Statistics assignment due for tonight in a few hours

Does anyone know how to do Linear Discriminant Analysis with more than one predictor - by hand? I know how to compute in R but my professor wants it done by hand .... fml

likefunny

Anyone have any suggestions for front end guis for users to run python code? I’ve created a few tools for webscraping data, automation, and modeling but non technical users freak out when they try to run the code themselves. Any suggestions? (Have tried Juypter Notebooks and a few other IDEs)

like

Business practice: "can you guys build a model that overlays where Coronavirus is going to break out with client sites"

likefunny

Has anyone else begun to resent data science?

like

Anyone have recommendations on the best app/site to learn Python? 🐍
I’m a complete beginner.

like

Anyone in the KPMG Lighthouse practice that would be willing to take a look at my husband’s resume and put in a referral? He’s interested in the Associate, Data Scientist position.

like

Are there RPA use cases in data work?

like

Any conda users in here have to install a binary and use conda forge? Need some help installing pdftotext. Throwing a weird message saying “Found conflicts!” Any and all help would be appreciated.

like

In the most unnecessarily complicated phrasing worthy a dissertation, what are you working on this week?

like

Newish to data analytics. Have some experience in power BI, sql, and R. I wanted to get some exposure to aws and saw the AWS cloud practitioner as an available course at my job. Is this cert worth it? Or should I try one of the more advanced ones ? The data analytics one for example but wasn’t sure if it would be difficult to get through as a beginner. Will it look good on my resume?

like

I have been working in healthcare clinics and health techs for a few years now. I’m an independent consultant. Bust, a friend of mine referred me to a big bank job. The role is of data governance manager. What should I know for the interview, or how can I prepare for it? Any thoughts? They told me I don’t need financial background. Data is data, but they wanted to know metadata management and how do I build frameworks/implement enterprise systems.

like

Anyone here work or has interviewed for a data science role at DataRobot? Would love to get some insight. Third interview coming up.

like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal