Apache Parquet Apache Parquet is a free and open-source column-oriented data storage format of the Apache Hadoop ecosystem. It is similar to the other columnar-storage file formats available in Hadoop namely RCFile and ORC . It is compatible with most of the data processing frameworks in the Hadoop environment. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk .History The open-source project to build Apache Parquet began as a joint effort between Twitter and Cloudera . Parquet was designed as an improvement upon the Trevni columnar storage format created by Hadoop creator Doug Cutting . The first version—Apache Parquet 1.0—was released in July 2013 . Since April 27 , 2015, Apache Parquet is a top-level Apache Software Foundation -sponsored project.Features Apache Parquet is implemented using the record-shredding and assembly algorithm, which accommodates the complex data structures that can be used to store the data. The values in each column are physically stored in contiguous memory locations and this columnar storage provides the following benefits: Column-wise compression is efficient and saves storage space Compression techniques specific to a type can be applied as the column values tend to be of the same type Queries that fetch specific column values need not read the entire row data thus improving performance Different encoding techniques can be applied to different columns Apache Parquet is implemented using the Apache Thrift framework which increases its flexibility; it can work with a number of programming languages like C++ , Java , Python , PHP , etc. As of August 2015 , Parquet supports the big-data-processing frameworks including Apache Hive , Apache Drill , Apache Impala , Apache Crunch , Apache Pig , Cascading , Presto and Apache Spark .Compression and encoding In Parquet, compression is performed column by column, which enables different encoding schemes to be used for text and integer data. This strategy also keeps the door open for newer and better encoding schemes to be implemented as they are invented.Dictionary encoding Parquet has an automatic dictionary encoding enabled dynamically for data with a small number of unique values that enables significant compression and boosts processing speed .Bit packing Storage of integers is usually done with dedicated 32 or 64 bits per integer. For small integers, packing multiple integers into the same space makes storage more efficient.To optimize storage of multiple occurrences of the same value, a single value is stored once along with the number of occurrences. Parquet implements a hybrid of bit packing and RLE, in which the encoding switches based on which produces the best compression results. This strategy works well for certain types of integer data and combines well with dictionary encoding.Comparison Apache Parquet is comparable to RCFile and Optimized Row Columnar file formats---all three fall under the category of columnar data storage within the Hadoop ecosystem. They all have better compression and encoding with improved read performance at the cost of slower writes. In addition to these features , Apache Parquet supports limited schema evolution , i.e., the schema can be modified according to the changes in the data. It also provides the ability to add new columns and merge schemas that don't conflict.
Popular articles Javier Milei - Argentine libertarian economist, author, radio conductor and public speaker sympathetic to the Austrian School of economic thought. He became widely known for his regular ...Jimmy Carter - American politician, philanthropist, and former farmer who served as the 39th president of the United States from 1977 to 1981. A member of the Democratic Party, he previ...UEFA Euro 2024 - The 2024 UEFA European Football Championship , commonly referred to as UEFA Euro 2024 or simply Euro 2024 , will be the 17th edition of the UEFA European Championship, the quadrennial internationa...Argentina - country located mostly in the southern half of South America. Sharing the bulk of the Southern Cone with Chile to the west, the country is also b...Sam Altman - American entrepreneur, investor, programmer, and blogger. He is the former president of Y Combinator and now the CEO of OpenAI. Early life and education. ...Rosalynn Carter - American who served as First Lady of the United States from 1977 to 1981 as the wife of President Jimmy Carter. For decades, she has been a leading advocate for numerou...Next Argentine presidential election - Next Argentine presidential election - presidential election in Argentina....Popular movies The Hunger Games (film) - 2012 American dystopian action thriller science fiction-adventure film directed by Gary Ross and based on Suzanne Collins’s 2008 novel of the same name. It is the first insta...untitled Captain Marvel sequel - part of Marvel Cinematic Universe....Killers of the Flower Moon (film project) - Killers of the Flower Moon - film project in United States of America. It was presented as drama, detective fiction, thriller. The film project starred Leonardo Dicaprio, Robert De Niro. Director of...Five Nights at Freddy's (film) - Five Nights at Freddy's - film published in 2017 in United States of America. Scenarist of the film - Scott Cawthon....Popular video games Minecraft - sandbox video game developed by Mojang Studios. Created by Markus "Notch" Persson in the Java programming language and released as a public alpha for personal computers in 2...Grand Theft Auto V - 2013 action-adventure game developed by Rockstar North and published by Rockstar Games. It is the first main entry in the Grand Theft Auto series since 2008's Grand Theft ...Roblox - online game platform and game creation system that allows users to program games and play games created by other users. Founded by David Baszucki and Erik Cassel in 2004 and released in...Baldur's Gate III - upcoming role-playing video game developed and published by Larian Studios for Microsoft Windows and the Stadia streaming service. It is the third main game in the Baldur's ...Alan Wake - action-adventure video game developed by Remedy Entertainment and published by Microsoft Studios, released for the Xbox 360 and Microsoft Windows. The story follows best-selling thri...Fortnite - online video game developed by Epic Games and released in 2017. It is available in three distinct game mode versions that otherwise share the same general gameplay and game engine: ...Super Mario RPG - is a role-playing video game developed by Square and published by Nintendo for the Super Nintendo Entertainment System in 1996. It was directed by Yoshihiko Maekawa and Chihiro Fujioka and produced by...Popular books Book of Revelation - The Book of Revelation is the final book of the New Testament, and consequently is also the final book of the Christian Bible. Its title is derived from the first word of the Koine Greek text: apok...Book of Genesis - account of the creation of the world, the early history of humanity, Israel's ancestors and the origins...Gospel of Matthew - The Gospel According to Matthew is the first book of the New Testament and one of the three synoptic gospels. It tells how Israel's Messiah, rejected and executed in Israel, pronounces judgement on ...Michelin Guide - Michelin Guides are a series of guide books published by the French tyre company Michelin for more than a century. The term normally refers to the annually published Michelin Red Guide , the oldest...Psalms - The Book of Psalms , commonly referred to simply as Psalms , the Psalter or "the Psalms", is the first book of the Ketuvim , the third section of the Hebrew Bible, and thus a book of th...Ecclesiastes - Ecclesiastes is one of 24 books of the Tanakh , where it is classified as one of the Ketuvim . Originally written c. 450–200 BCE, it is also among the canonical Wisdom literature of the Old Tes...The 48 Laws of Power - non-fiction book by American author Robert Greene. The book...Popular television series The Crown (TV series) - historical drama web television series about the reign of Queen Elizabeth II, created and principally written by Peter Morgan, and produced by Left Bank Pictures and Sony Pictures Tel...Friends - American sitcom television series, created by David Crane and Marta Kauffman, which aired on NBC from September 22, 1994, to May 6, 2004, lasting ten seasons. With an ensemble cast sta...Young Sheldon - spin-off prequel to The Big Bang Theory and begins with the character Sheldon...Modern Family - American television mockumentary family sitcom created by Christopher Lloyd and Steven Levitan for the American Broadcasting Company. It ran for eleven seasons, from September 23...Loki (TV series) - upcoming American web television miniseries created for Disney+ by Michael Waldron, based on the Marvel Comics character of the same name. It is set in the Marvel Cinematic Universe, shar...Game of Thrones - American fantasy drama television series created by David Benioff and D. B. Weiss for HBO. It...Shameless (American TV series) - American comedy-drama television series developed by John Wells which debuted on Showtime on January 9, 2011. It...
OWIKI.org . Text is available under the Creative Commons Attribution-ShareAlike License.