Skip to content

@hplt-project HPLT - High Performance Language Technologies

Change the repository type filter

All

    Repositories list

    37 repositories

    • HPLT-textpipes

      Public
      Step-by-step schematic description of data processing in HPLT
      Python
      •
      MIT License
      •1•1•2•0•Updated Apr 19, 2026Apr 19, 2026

    • HPLT-WP4

      Public
      Information and pipelines on WP4: language models training
      Jupyter Notebook
      •
      Creative Commons Zero v1.0 Universal
      •3•3•5•0•Updated Mar 30, 2026Mar 30, 2026

    • bitextor-slurm

      Public
      Scripts for running bitextor jobs
      Shell
      •1•0•1•0•Updated Mar 15, 2026Mar 15, 2026

    • openlid-v3-evaluation

      Public
      OpenLID-v3
      language-identification
      language-identification
      HTML
      •0•1•0•0•Updated Mar 9, 2026Mar 9, 2026

    • monotextor-slurm

      Public
      Set of scripts to run monotextor-like pipeline under slurm HPCs
      Rust
      •
      GNU General Public License v3.0
      •1•5•0•0•Updated Feb 26, 2026Feb 26, 2026

    • data-analytics-tool

      Public
      HPLT Analytics
      JavaScript
      •
      GNU General Public License v3.0
      •4•15•0•1•Updated Feb 23, 2026Feb 23, 2026

    • openlid

      Public
      OpenLID-v3
      Python
      •
      GNU General Public License v3.0
      •5•1•0•0•Updated Feb 20, 2026Feb 20, 2026

    • monolingual-multilingual-instruction-tuning

      Public
      Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca
      large-language-modelsinstruction-tuninginstruction-data
      large-language-modelsinstruction-tuninginstruction-data
      Python
      •1•10•0•0•Updated Feb 16, 2026Feb 16, 2026

    • hplt-e

      Public
      Multilingual evaluation framework
      Python
      •
      MIT License
      •0•3•0•0•Updated Feb 11, 2026Feb 11, 2026

    • sacremoses

      Public
      Python port of Moses tokenizer, truecaser and normalizer
      nlptokenizermachine-translation
      nlptokenizermachine-translation
      Python
      •
      MIT License
      •59•494•28•4•Updated Feb 6, 2026Feb 6, 2026

    • clianer

      Public
      A lightweight command-line frontend to OpusCleaner
      Python
      •
      MIT License
      •1•0•0•0•Updated Feb 5, 2026Feb 5, 2026

    • OpusCleaner

      Public
      OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.
      machine-translationdata-cleaning
      machine-translationdata-cleaning
      Python
      •16•58•56•1•Updated Feb 3, 2026Feb 3, 2026

    • warc2text-runner

      Public
      Scripts for parallelized extraction of plain texts from WARC archieves. Aiming at common and reproducible extraction approach.
      Jupyter Notebook
      •
      MIT License
      •1•4•0•0•Updated Jan 31, 2026Jan 31, 2026

    • Megatron-LM

      Public
      Ongoing research training transformer models at scale
      Python
      •
      Other
      •3.9k•0•0•0•Updated Jan 28, 2026Jan 28, 2026

    • OpusPocus

      Public
      Marian machine translation training pipeline for thousands of models
      Python
      •
      MIT License
      •0•4•16•0•Updated Jan 27, 2026Jan 27, 2026

    • release2_inspection

      Public
      Jupyter Notebook
      •
      MIT License
      •7•1•1•0•Updated Jan 26, 2026Jan 26, 2026

    • release3_inspection

      Public
      Jupyter Notebook
      •
      MIT License
      •9•0•0•0•Updated Jan 26, 2026Jan 26, 2026

    • document-aligner

      Public
      tf/idf-based document aligner from Bitextor
      C++
      •
      Apache License 2.0
      •0•0•0•1•Updated Jan 22, 2026Jan 22, 2026

    • bitextor-mt-models

      Public
      Shell
      •0•1•3•0•Updated Jan 22, 2026Jan 22, 2026

    • ia-download

      Public
      Internet archive downloader
      Jupyter Notebook
      •
      MIT License
      •0•2•2•0•Updated Jan 22, 2026Jan 22, 2026

    • minhash-retrieval

      Public
      Find your documents in the HPLT datasets
      Python
      •
      MIT License
      •1•0•2•0•Updated Jan 22, 2026Jan 22, 2026

    • OPUS-MT-app

      Public
      Fast and secure translation on your local machine, powered by marian and Bergamot.
      C++
      •
      MIT License
      •40•0•0•0•Updated Jan 21, 2026Jan 21, 2026

    • OpusExplorer

      Public
      web interface for exploring OPUS data
      PHP
      •1•0•0•0•Updated Jan 21, 2026Jan 21, 2026

    • OPUS-API

      Public
      API for searching corpora from OPUS
      Python
      •2•0•0•0•Updated Jan 21, 2026Jan 21, 2026

    • OPUS-ingest

      Public
      Makefile
      •3•0•0•0•Updated Jan 21, 2026Jan 21, 2026

    • OpusTools

      Public
      Python
      •22•0•0•0•Updated Jan 21, 2026Jan 21, 2026

    • OPUS

      Public
      The Open Parallel Corpus
      Makefile
      •14•0•0•0•Updated Jan 21, 2026Jan 21, 2026

    • OPUS-MT-dashboard

      Public
      PHP
      •
      MIT License
      •1•0•0•0•Updated Jan 21, 2026Jan 21, 2026

    • OpusFilter

      Public
      OpusFilter - Parallel corpus processing toolkit
      Python
      •
      MIT License
      •26•0•0•0•Updated Jan 21, 2026Jan 21, 2026

    • OpusDistillery

      Public
      Training pipelines for Firefox Translations neural machine translation models (adapted for OPUS-MT and integrating GreenNLP metrics)
      Python
      •
      Mozilla Public License 2.0
      •46•0•0•0•Updated Jan 8, 2026Jan 8, 2026
    ProTip! When viewing an organization's repositories, you can use the props. filter to filter by custom property.