From db4882a382392548633c490d2bea13aef46d8a8d Mon Sep 17 00:00:00 2001 From: zoltanvb Date: Sun, 25 Jan 2026 12:43:52 +0100 Subject: [PATCH] Adjustments related to scan rework Updated instructions in line with the new Loose scan option. References to "Scan Directory", "Scan File", "Manual scan" and other menu items are updated to reflect the current UI. --- docs/guides/arcade-getting-started.md | 2 +- docs/guides/databases.md | 276 +++++++++++------------ docs/guides/import-content.md | 14 +- docs/guides/roms-playlists-thumbnails.md | 8 +- 4 files changed, 144 insertions(+), 156 deletions(-) diff --git a/docs/guides/arcade-getting-started.md b/docs/guides/arcade-getting-started.md index e05e4edbf4..8850ee51d1 100644 --- a/docs/guides/arcade-getting-started.md +++ b/docs/guides/arcade-getting-started.md @@ -253,7 +253,7 @@ Time to find out how well your source ROMs matched up... ~~The RetroArch content database supports arcade romsets in Full Non-Merged and Split formats. In order to be recognized by the scanner, Full Non-Merged and Split romsets must also be [processed by TorrentZip to standardize their CRC](https://sourceforge.net/projects/trrntzip/).~~ -Manual scan is the recommended method for setting up arcade playlists in RetroArch. [Here](https://neo-source.com/index.php?topic=3725.0) is a guide written for creating FBNeo playlists, but you can easily adapt it for MAME usage. +Custom scan with DAT file is the recommended method for setting up arcade playlists in RetroArch. [Here](https://neo-source.com/index.php?topic=3725.0) is a guide written for creating FBNeo playlists, but you can easily adapt it for MAME usage. !!! info "Credits" The arcade cabinets image is based on an image by Rob DiCaterino, licensed for reuse under a Creative Commons (CC BY 2.0) License. Original image and license: https://www.flickr.com/photos/goodrob13/17385639015/ diff --git a/docs/guides/databases.md b/docs/guides/databases.md index 4857c14309..72ee215a91 100644 --- a/docs/guides/databases.md +++ b/docs/guides/databases.md @@ -1,139 +1,137 @@ -# Databases - -RetroArch uses `.rdb` [database format](https://github.com/libretro/RetroArch/blob/master/libretro-db/README.md) files stored locally by default in folder `RetroArch/databases`. The `.rdb` files are compiled from clrmamepro format `.dat` files stored at the [libretro database repository](https://github.com/libretro/libretro-database). See the database [readme](https://github.com/libretro/libretro-database/blob/master/README.md) for comprehensive information about the sources and functioning of the repository. - -!!! Hint "Terminology Note: Game Name" - The term _Game Name_ refers to the name displayed [within the RetroArch interface and in playlists](https://docs.libretro.com/guides/roms-playlists-thumbnails/#retroarch-playlist-scanner), _not_ to the filename of the underlying file on the computer or device. _Game Name_ in this document is synonymous with playlist item label, playlist entry, content name, and game title. - -## Features of RetroArch Database Usage - -RetroArch uses the database to provide several automated cataloging functions: - -- __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum (or [other key](#key-field-for-matching)) matches the checksum of a known verified completely intact (aka "properly dumped") file in the database. -- __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. RetroArch will look up the `name` field specified for the file's [key](#key-field-for-matching) in the database. - - _Secondary_: __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Thumbnails are __not__ directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server. Also see: [Flexible Name Matching Algorithm](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails).) -- __Category Search ("Explore")__. Allows the user to find/view games that match selected criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. -- __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry). - -## How the Import Scanner Uses the Database - -_See also: [Importing Content](https://docs.libretro.com/guides/import-content/) and [Creating Playlists](https://docs.libretro.com/guides/roms-playlists-thumbnails/#retroarch-playlist-scanner)._ - -RetroArch's Import Content actions "Directory Scan" and "File Scan" will do the following: - -1. Compute a CRC checksum of the file(s) or scan for the in-game serial number. CRC and serial number are the [keys used for matching a game file to the database](#key-field-for-matching). -1. Search for that CRC or serial in the information of the local `.rdb` files (default location `RetroArch/databases`). If the key is not found in databases, the file will __not__ be added to a playlist. See [Validation & Rejection](#validation-and-rejection). -3. Assign the `Game Name` (aka display name or playlist item name) that is specified as `name` within the database entry for the key (CRC/serial). The assigned `Game Name` will appear in the playlist, instead of the filename. -4. All other associated metadata [collated in the .rdb](https://github.com/libretro/libretro-database#fields-specified-in-game-information-databases) entry for the given CRC/serial can be viewed in the Information > Database Entry for the game and will be viewable via "Explore". - -### Validation and Rejection - -Validation here refers to checking a file or attribute against a reference, and then accepting or rejecting it based on whether it matches what is specified in the reference data (aka what is "allowed"). RetroArch's "Scan Directory" and "Scan File" automated importers are validation processes, not merely tools for adding all files to a playlist. Part of their function is to **reject** files, not to import all files. The database is the reference, and the ROM file is the item being validated. - -If your file's crc or internal serial data (whichever is the key used for matching, [see below](#key-field-for-matching)) does not exist in the database, the file will be rejected by the automatic scans and will not appear in the playlist. - -__Bypass validation and rejection.__ To import your games into a playlist regardless of database matches, or if your files are being rejected by the automatic scan (in other words are not recognized by the database) and you wish to add them to the playlist anyway, use the [Manual Scan](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists). - -### Key Field for Matching - -During [Playlist / Import Scanning](https://docs.libretro.com/guides/roms-playlists-thumbnails/#retroarch-playlist-scanner) ("Directory Scan" and "Scan File" in menu), RetroArch will identify your _files_ in order to then match your file to a data entry in the database. The key for matching varies by console typical file size (i.e. original media type). - -- __CRC checksum__ for systems with smaller file sizes, e.g. games before the advent of disc-based consoles. -- __Serial Number__ for larger files like disc-based games, to avoid computing checksums on large files. Found within the ROM file. The serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. - -Contrary to popular belief, the data used for matching is often the _serial number_ encoded within a disc-game's binary data. - -Databases include cryptographic hashes (sha1, etc) for informational purposes to define the item specified, but only CRC checksum (or serial) not hashes are used for matching. - -## Databases and Thumbnails - -Thumbnails _are not_ assigned or retrieved based on checksum, serial, or game database matching. See separate documentation for [thumbnail handling](https://docs.libretro.com/guides/roms-playlists-thumbnails/#thumbnails) and the thumbnail [matching algorithm](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails). - -Currently there is no _automatic_ process that applies database game name changes/updates to libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails) image filenames. Therefore one of the visible consequences of a Game Name or database problem is the lack of an appropriate thumbnail display in RetroArch whenever the `Game Name` displayed in the interface doesn't match a repository thumbnail filename. - -- __Game name error__. To help fix a database error where the game name doesn't match a correctly named thumbnail in the repository, see [How to Contribute to Databases](#how-to-contribute-to-databases). -- __Thumbnail name error__. To help fix a thumbnail in a case where a _correct_ database game name doesn't match the repository thumbnail name, follow the [Thumbnail Repository readme](https://github.com/libretro-thumbnails/libretro-thumbnails/blob/master/README.md#contributions) and [How To Contribute Thumbnails guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). - - -## Troubleshooting - -_See also: RetroArch documentation for [Import Scanning](https://docs.libretro.com/guides/import-content/) and [Playlist Creation/Scanning](https://docs.libretro.com/guides/roms-playlists-thumbnails/#retroarch-playlist-scanner)._ - -The information below is for Users who are interested in figuring out the cause of a database-related problem and possibly helping to fix the problem in a way that will help all users. RetroArch is a volunteer project, and many problem situations can be improved by interested users with medium technical awareness and no programming skill needed. - -### Common Problems and Solutions - -The most common user problems and solutions related to the database are: - -- __Missed files during import scan.__ I.e. automated Directory Scan or File Scan "misses" some files, meaning the files are not imported and do not appear in the playlist. See [Validation & Rejection](#validation-and-rejection) above. - - Solution A: [Contribute to the database](#how-to-contribute-to-databases) with data for the files/games that are not yet covered by the database. - - Solution B: Use the __Manual Scan__ option, which will accept all files according to the chosen settings. -- __Game Name error or incorrect information__. E.g. A game file receives a wrong title inside the RetroArch playlist/interface. - - Solution: - - Follow the [investigation steps](#investigating-database-issues) below to find the `.dat` file that has the erroneous information, and [contribute a correction](#how-to-contribute-to-databases). - - Depending on the source of the data, an upstream change within a database group's system may be required, but it is also possible to create ad hoc database coverage on the libretro github. -- __Outdated Local Files__. I.e. an error(s) has been fixed in the libretro database but the fix has not yet been downloaded in the user's app install. - - Solution: Update your RetroArch databases (Main Menu > Online Updater > Update databases). That will apply recent fixes/corrections to your RetroArch install. - -### Investigating Database Issues - -Follow the steps below to find the cause of a database or game/name identification issue: - -- __Learn about the factors that might be causing the Problem__ - - Understand the multifaceted [.dat system](https://github.com/libretro/libretro-database#retroarch-database) and files. Multiple different .dat files may have data for the same game. - - Understand [which key field](#key-field-for-matching) RetroArch uses for identifying your file and for searching for your file's info in the database. - - Understand [precedence](https://github.com/libretro/libretro-database#precedence) within the dat files in the repository. -- __Verify data "on both sides"__ - - __Verify your file properties.__ Verify your game file has the appropriate [key ID](#key-field-for-matching): compute the crc checksum, or verify the encoded serial number with a hex editor, whichever is applicable. - - __Verify libretro databases on github__. Look in the repository databases to find which `.dat` file might hold incorrect data for the game file at issue. Even if you find a `.dat` that holds correct data for the game key (CRC or serial) in question, a different dat with [precedence](https://github.com/libretro/libretro-database#precedence) may hold incorrect data that is over-ruling the correct data. -- __Verify upstream data__ to find mismatches or pending corrections. If an upstream database group (No-Intro, Redump, GameTDB, etc) is [responsible for the `.dat` at issue](https://github.com/libretro/libretro-database#sources), look on their websites to see whether their current information is correct or incorrect. - -### Help Fix the Problem - -After you've investigated the issue (see above), some possible actions are: - -- __Update__. Use _Main Menu > Online Updater > Update Databases_ to download new and possibly corrected data from the libretro server. -- __[Contribute](#how-to-contribute-to-databases)__ a correction or addition of data to fix the issue. It may be possible to create an ad hoc database or to make a new entry within an existing ad hoc database. -- __Use the [Issue Tracker](https://github.com/libretro/libretro-database/issues)__. - - __Search__ for existing issues on github that may hold useful advice or solutions for your problem. Adding your new examples or insights to the discussion for the problem may help Members/Contributors create a fix. - - __Open__ an Issue if a relevant one isn't already open. - - Open a [Database Issue](https://github.com/libretro/libretro-database/issues) __if__ you observe either of the following: - - You see a large-scale issue affecting many data entries or entire dats. - - You found that Upstream Data is _correct_ but libretro or RetroArch doesn't reflect it, and at leat 4 weeks have passed since the Upstream update occurred. - - Open a [RetroArch Issue](https://github.com/libretro/RetroArch/issues) __if__: you see a problem with RetroArch's scanning behavior or validation, while the databases appear correct and match your file's properties (crc and serial within the game's binary data viewable with a hex editor). -- __Submit Upstream Changes.__ Make changes upstream (No-Intro, Redump, GameTDB, etc) by going through the channels of the upstream group responsible for the data at issue __if__: you found that Upstream Data is _Incorrect_ and has been imported to the libretro database repository. The upstream group must make the correction to "fix it at the source", though it may be possible to create alternative data coverage instead (see below). - -## How to Contribute to Databases - -Like [thumbnails](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to) and [documentation](https://docs.libretro.com/meta/how-to-contribute/), databases are an area where users who are not programmers can contribute to RetroArch and in a way that benefits all users. - -### Github Steps Overview - -1. Make an account on github.com and login. -2. Fork the libretro database repository. Forking means copying the repository into your own working area on github. -3. Make your changes within your fork. For example, create a new `.dat`, or add a game entry or correction to an existing dat. - - Understand the principles and [header specifications](https://github.com/libretro/libretro-database#header-guidelines-for-dats) explained in the [database repository readme](https://github.com/libretro/libretro-database#). - - Carefully heed the existing formats observable in present `.dat` files on the repository. -5. "Commit" (save) your changes with your fork. -6. Use the "Pull Request" button (or the Contribute > Open Pull Request button) to send and propose your changes to the libretro official team members. They will review your contribution, and in time either accept it or inform you about required changes or a reason why it isn't acceptable. - -### Small-Scale Corrections - -See [database repository readme](https://github.com/libretro/libretro-database#small-scale-corrections) to learn when/where small-scare corrections are appropriate. - -Two methods for adding data coverage for a single game or niche of games, via Pull Request proposal on github: - -- __Method A:__ Fix the dat at issue. This is only possible if two conditions are met: - - 1. The `.dat` doesn't originate from an import from upstream _and_ - - 2. The `.dat` won't receive subtractive sync over-writes in the future. - - _If those conditions are not met, the intended "fix" would be deleted by the next bulk import sync._ - - _Or..._ - -- __Method B:__ Edit a different dat, leaving the erroneous dat intact but moot. This is only advisable when the correction and the error have different [keys](#key-field-for-matching), or if the edited/corrected database has [precedence](https://github.com/libretro/libretro-database#precedence) over the erroneous database. If one of those conditions is _not_ met, then the attempted correction would fail: it would be over-ruled in the `.rdb` compile by the erroneous dat's information. If one of those conditions is met, you may do one of following: - - Add a game data entry to an existing and appropriate ad hoc `.dat` in the repository. - - Create a new ad hoc `.dat`. This is often acceptable even for a small number of games because of the multi-faceted nature of the dat system. Some limitations may be enforced by admins, e.g. for the manageability of the build script or the repository. - -### Large-Scale Additions - -See [Adding New Database](https://github.com/libretro/libretro-database#adding-a-new-database). Contributors are welcome to propose the addition of bulk data from their own build scripts or otherwise, via github Pull Request. +# Databases + +RetroArch uses `.rdb` [database format](https://github.com/libretro/RetroArch/blob/master/libretro-db/README.md) files stored locally by default in folder `RetroArch/databases`. The `.rdb` files are compiled from clrmamepro format `.dat` files stored at the [libretro database repository](https://github.com/libretro/libretro-database). See the database [readme](https://github.com/libretro/libretro-database/blob/master/README.md) for comprehensive information about the sources and functioning of the repository. + +!!! Hint "Terminology Note: Game Name" + The term _Game Name_ refers to the name displayed [within the RetroArch interface and in playlists](https://docs.libretro.com/guides/roms-playlists-thumbnails/#retroarch-playlist-scanner), _not_ to the filename of the underlying file on the computer or device. _Game Name_ in this document is synonymous with playlist item label, playlist entry, content name, and game title. + +## Features of RetroArch Database Usage + +RetroArch uses the database to provide several automated cataloging functions: + +- __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum (or [other key](#key-field-for-matching)) matches the checksum of a known verified completely intact (aka "properly dumped") file in the database. +- __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. RetroArch will look up the `name` field specified for the file's [key](#key-field-for-matching) in the database. + - _Secondary_: __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Thumbnails are __not__ directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server. Also see: [Flexible Name Matching Algorithm](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails).) +- __Category Search ("Explore")__. Allows the user to find/view games that match selected criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. +- __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry). + +## How the Import Scanner Uses the Database + +_See also: [Importing Content](https://docs.libretro.com/guides/import-content/) and [Creating Playlists](https://docs.libretro.com/guides/roms-playlists-thumbnails/#retroarch-playlist-scanner)._ + +RetroArch's Import Content actions will do the following if strict (default) or loose database check is used: + +1. Compute a CRC checksum of the file(s) or scan for the in-game serial number. CRC and serial number are the [keys used for matching a game file to the database](#key-field-for-matching). +1. Search for that CRC or serial in the information of the local `.rdb` files (default location `RetroArch/databases`). If the key is not found in databases, the file will __not__ be added to a playlist in strict mode. See [Validation & Rejection](#validation-and-rejection). +3. Assign the `Game Name` (aka display name or playlist item name) that is specified as `name` within the database entry for the key (CRC/serial). The assigned `Game Name` will appear in the playlist, instead of the filename. +4. All other associated metadata [collated in the .rdb](https://github.com/libretro/libretro-database#fields-specified-in-game-information-databases) entry for the given CRC/serial can be viewed in the Information > Database Entry for the game and will be viewable via "Explore". + +### Validation and Rejection + +Validation here refers to checking a file or attribute against a reference, and then accepting or rejecting it based on whether it matches what is specified in the reference data (aka what is "allowed"). RetroArch's strict scan is also a validation process, not merely a tool for adding all files to a playlist. Part of the function is to **reject** certain files. The database is the reference, and the ROM file is the item being validated. + +If your file's crc or internal serial data (whichever is the key used for matching, [see below](#key-field-for-matching)) does not exist in the database, the file will be rejected by the strict scans and will not appear in the playlist. By changing the Database Check parameter to Loose, such files will still be added to playlist. + +### Key Field for Matching + +During [Playlist / Import Scanning](https://docs.libretro.com/guides/roms-playlists-thumbnails/#retroarch-playlist-scanner), RetroArch will identify your _files_ in order to then match your file to a data entry in the database. The key for matching varies by console original media type. + +- __CRC checksum__ for systems with smaller file sizes, e.g. games before the advent of disc-based consoles. +- __Serial Number__ for larger files like disc-based games, to avoid computing checksums on large files. Found within the ROM file. The serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. + +Contrary to popular belief, the data used for matching is often the _serial number_ encoded within a disc-game's binary data. + +Databases include cryptographic hashes (sha1, etc) for informational purposes to define the item specified, but only CRC checksum (or serial) not hashes are used for matching. + +## Databases and Thumbnails + +Thumbnails _are not_ assigned or retrieved based on checksum, serial, or game database matching. See separate documentation for [thumbnail handling](https://docs.libretro.com/guides/roms-playlists-thumbnails/#thumbnails) and the thumbnail [matching algorithm](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails). + +Currently there is no _automatic_ process that applies database game name changes/updates to libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails) image filenames. Therefore one of the visible consequences of a Game Name or database problem is the lack of an appropriate thumbnail display in RetroArch whenever the `Game Name` displayed in the interface doesn't match a repository thumbnail filename. + +- __Game name error__. To help fix a database error where the game name doesn't match a correctly named thumbnail in the repository, see [How to Contribute to Databases](#how-to-contribute-to-databases). +- __Thumbnail name error__. To help fix a thumbnail in a case where a _correct_ database game name doesn't match the repository thumbnail name, follow the [Thumbnail Repository readme](https://github.com/libretro-thumbnails/libretro-thumbnails/blob/master/README.md#contributions) and [How To Contribute Thumbnails guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). + + +## Troubleshooting + +_See also: RetroArch documentation for [Import Scanning](https://docs.libretro.com/guides/import-content/) and [Playlist Creation/Scanning](https://docs.libretro.com/guides/roms-playlists-thumbnails/#retroarch-playlist-scanner)._ + +The information below is for Users who are interested in figuring out the cause of a database-related problem and possibly helping to fix the problem in a way that will help all users. RetroArch is a volunteer project, and many problem situations can be improved by interested users with medium technical awareness and no programming skill needed. + +### Common Problems and Solutions + +The most common user problems and solutions related to the database are: + +- __Missed files during import scan.__ I.e. default strict scan "misses" some files, meaning the files are not imported and do not appear in the playlist. See [Validation & Rejection](#validation-and-rejection) above. + - Solution A: [Contribute to the database](#how-to-contribute-to-databases) with data for the files/games that are not yet covered by the database. + - Solution B: Use the __Loose__ option, which will accept all files according to the chosen settings. +- __Game Name error or incorrect information__. E.g. A game file receives a wrong title inside the RetroArch playlist/interface. + - Solution: + - Follow the [investigation steps](#investigating-database-issues) below to find the `.dat` file that has the erroneous information, and [contribute a correction](#how-to-contribute-to-databases). + - Depending on the source of the data, an upstream change within a database group's system may be required, but it is also possible to create ad hoc database coverage on the libretro github. +- __Outdated Local Files__. I.e. an error(s) has been fixed in the libretro database but the fix has not yet been downloaded in the user's app install. + - Solution: Update your RetroArch databases (Main Menu > Online Updater > Update databases). That will apply recent fixes/corrections to your RetroArch install. + +### Investigating Database Issues + +Follow the steps below to find the cause of a database or game/name identification issue: + +- __Learn about the factors that might be causing the Problem__ + - Understand the multifaceted [.dat system](https://github.com/libretro/libretro-database#retroarch-database) and files. Multiple different .dat files may have data for the same game. + - Understand [which key field](#key-field-for-matching) RetroArch uses for identifying your file and for searching for your file's info in the database. + - Understand [precedence](https://github.com/libretro/libretro-database#precedence) within the dat files in the repository. +- __Verify data "on both sides"__ + - __Verify your file properties.__ Verify your game file has the appropriate [key ID](#key-field-for-matching): compute the crc checksum, or verify the encoded serial number with a hex editor, whichever is applicable. + - __Verify libretro databases on github__. Look in the repository databases to find which `.dat` file might hold incorrect data for the game file at issue. Even if you find a `.dat` that holds correct data for the game key (CRC or serial) in question, a different dat with [precedence](https://github.com/libretro/libretro-database#precedence) may hold incorrect data that is over-ruling the correct data. +- __Verify upstream data__ to find mismatches or pending corrections. If an upstream database group (No-Intro, Redump, GameTDB, etc) is [responsible for the `.dat` at issue](https://github.com/libretro/libretro-database#sources), look on their websites to see whether their current information is correct or incorrect. + +### Help Fix the Problem + +After you've investigated the issue (see above), some possible actions are: + +- __Update__. Use _Main Menu > Online Updater > Update Databases_ to download new and possibly corrected data from the libretro server. +- __[Contribute](#how-to-contribute-to-databases)__ a correction or addition of data to fix the issue. It may be possible to create an ad hoc database or to make a new entry within an existing ad hoc database. +- __Use the [Issue Tracker](https://github.com/libretro/libretro-database/issues)__. + - __Search__ for existing issues on github that may hold useful advice or solutions for your problem. Adding your new examples or insights to the discussion for the problem may help Members/Contributors create a fix. + - __Open__ an Issue if a relevant one isn't already open. + - Open a [Database Issue](https://github.com/libretro/libretro-database/issues) __if__ you observe either of the following: + - You see a large-scale issue affecting many data entries or entire dats. + - You found that Upstream Data is _correct_ but libretro or RetroArch doesn't reflect it, and at leat 4 weeks have passed since the Upstream update occurred. + - Open a [RetroArch Issue](https://github.com/libretro/RetroArch/issues) __if__: you see a problem with RetroArch's scanning behavior or validation, while the databases appear correct and match your file's properties (crc and serial within the game's binary data viewable with a hex editor). +- __Submit Upstream Changes.__ Make changes upstream (No-Intro, Redump, GameTDB, etc) by going through the channels of the upstream group responsible for the data at issue __if__: you found that Upstream Data is _Incorrect_ and has been imported to the libretro database repository. The upstream group must make the correction to "fix it at the source", though it may be possible to create alternative data coverage instead (see below). + +## How to Contribute to Databases + +Like [thumbnails](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to) and [documentation](https://docs.libretro.com/meta/how-to-contribute/), databases are an area where users who are not programmers can contribute to RetroArch and in a way that benefits all users. + +### Github Steps Overview + +1. Make an account on github.com and login. +2. Fork the libretro database repository. Forking means copying the repository into your own working area on github. +3. Make your changes within your fork. For example, create a new `.dat`, or add a game entry or correction to an existing dat. + - Understand the principles and [header specifications](https://github.com/libretro/libretro-database#header-guidelines-for-dats) explained in the [database repository readme](https://github.com/libretro/libretro-database#). + - Carefully heed the existing formats observable in present `.dat` files on the repository. +5. "Commit" (save) your changes with your fork. +6. Use the "Pull Request" button (or the Contribute > Open Pull Request button) to send and propose your changes to the libretro official team members. They will review your contribution, and in time either accept it or inform you about required changes or a reason why it isn't acceptable. + +### Small-Scale Corrections + +See [database repository readme](https://github.com/libretro/libretro-database#small-scale-corrections) to learn when/where small-scare corrections are appropriate. + +Two methods for adding data coverage for a single game or niche of games, via Pull Request proposal on github: + +- __Method A:__ Fix the dat at issue. This is only possible if two conditions are met: + - 1. The `.dat` doesn't originate from an import from upstream _and_ + - 2. The `.dat` won't receive subtractive sync over-writes in the future. + - _If those conditions are not met, the intended "fix" would be deleted by the next bulk import sync._ + + _Or..._ + +- __Method B:__ Edit a different dat, leaving the erroneous dat intact but moot. This is only advisable when the correction and the error have different [keys](#key-field-for-matching), or if the edited/corrected database has [precedence](https://github.com/libretro/libretro-database#precedence) over the erroneous database. If one of those conditions is _not_ met, then the attempted correction would fail: it would be over-ruled in the `.rdb` compile by the erroneous dat's information. If one of those conditions is met, you may do one of following: + - Add a game data entry to an existing and appropriate ad hoc `.dat` in the repository. + - Create a new ad hoc `.dat`. This is often acceptable even for a small number of games because of the multi-faceted nature of the dat system. Some limitations may be enforced by admins, e.g. for the manageability of the build script or the repository. + +### Large-Scale Additions + +See [Adding New Database](https://github.com/libretro/libretro-database#adding-a-new-database). Contributors are welcome to propose the addition of bulk data from their own build scripts or otherwise, via github Pull Request. diff --git a/docs/guides/import-content.md b/docs/guides/import-content.md index dcbf149d7a..385e15a3dc 100644 --- a/docs/guides/import-content.md +++ b/docs/guides/import-content.md @@ -12,7 +12,7 @@ In order to have RetroArch recognise your games, you need to have a database of !!! note Certain releases of Retroarch (e.g. Windows), come with the database files out-of-the-box. -From the RetroArch main menu select "Online Updater", then choose "Update Databases". +From the RetroArch main menu select "Online Updater", then choose "Update Databases". Also update core info files. Wait for the download to finish. @@ -23,17 +23,7 @@ The "Import Content" menu can be either in the "Playlists" > "Import Content" (d !!! note The location of this menu item can be toggled in "Settings" > "User Interface" > "Menu Item Visibility" > "Show 'Import Content'" and updating this from "Playlists Menu" to "Main Menu". -Here you will see three options, "Scan Directory", "Scan File" and "Manual Scan". - -!!! note - "Scan Directory" and "Scan File" are sometimes refered to as "Auto Scan". These options can only recognise content that match the database. - Using the default settings, these scan options also need a coresponding core for the said content to be already added. - -***1) Scan Directory:*** for importing a collection of content. Using the file browser, navigate to the folder of the content collection and select **"Scan This Directory"**. - -***2) Scan File:*** for importing a single file. Navigate to file and select it. - -***3) Manual Scan:*** it scans based on content file names and does not require content to match the database. +The default, fully automatic option for scanning is strict (requires the CRC checksum or disc serial of the content to match the database). Scanning can be customized in several ways, loose scans for example will add content that does not match the database. Please be patient while it scans. Large collections could take several minutes. diff --git a/docs/guides/roms-playlists-thumbnails.md b/docs/guides/roms-playlists-thumbnails.md index e96e09c4f8..30e5ba004e 100644 --- a/docs/guides/roms-playlists-thumbnails.md +++ b/docs/guides/roms-playlists-thumbnails.md @@ -30,16 +30,16 @@ Playlists are the lists of games and other content that can be generated automat RetroArch incorporates a ROM scanning system to automatically produce playlists. Each ROM that is scanned by the playlist generator is checked against a database of ROMs that are known to be good copies. -In order for content to be correctly scanned, you must: +To ensure that scanning can use all available information: - - Have a compatible core already downloaded and installed (note: Scan Without Core Match setting removes this requirement) + - Have a compatible core already downloaded and installed (or enable Scan Without Core Match during scanning) - Update `Core Info Files` via `Online Updater` - Update `Databases` via `Online Updater` - Restart RetroArch if any of the above was just done -For a normal scan, the content must match existing databases from the [libretro-database README](https://github.com/libretro/libretro-database#retroarch-database). If those conditions are met but content is still not being added automatically to a playlist, consider submitting an issue report on [github](https://www.github.com/libretro/RetroArch/issues). +The default fully automatic scan is strict, the content CRC checksum or disc serial must match existing databases from the [libretro-database](https://github.com/libretro/libretro-database?tab=readme-ov-file#libretro-database). If those conditions are met but content is still not being added automatically to a playlist, consider submitting an issue report on [github](https://www.github.com/libretro/RetroArch/issues). -There is an option to do manual scan, which does not require a database, and just needs the file names to match. Results from the manual scan will be playable (as long as the respective core supports them), but may lack thumbnails and do not appear in the Explore menu. +Scan conditions could be relaxed ("loose scan"). If there is no database match, the content is still playable (as long as the respective core supports them), but may lack thumbnails and do not appear in the Explore menu. ### Designating which core to use