Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

the portable Python dataframe library

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Lightweight and extensible compatibility layer between dataframe libraries!

Command-line interface to quickly generate fake CSV and JSON data

Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features

An open-source tool for reading OvertureMaps data with multiprocessing and additional Quality-of-Life features

db2ixf is a python package with a CLI that simplifies the parsing and processing of IBM Integration eXchange Format (IXF) files.

Type annotations for pyarrow

Poor mans simple python api for creating a local or remote datalake based on several (pyarrow) datasets using duckdb

Seamlessly switch Pandas DataFrame backend to PyArrow.

highspeed timeseries pandas dataframe database

A loose implementation of the deltalake protocol, written in Python on top of pyarrow, focused on extensibility, customizability, and distributed data.

Dremio Arrow Flight Client

Python scripts to process, and analyze log files using PySpark.

poor man´s data lake - Simple api to efficiently query your parquet datasets using Duckdb or polars

Concise interface to cache numpy arrays and pandas dataframes

View Apache Parquet Files In Your Terminal

A bioinformatics extension of 🤗 Datasets library, built for ML applications on biological and omics data, offering easy integration of metadata and low-code data management tools.

Code examples / snippets for website news post