fdroidserver.common module

class fdroidserver.common.ClonedZipInfo(zinfo)

Bases: ZipInfo

Hack to allow fully cloning ZipInfo instances.

The zipfile library has some bugs that prevent it from fully cloning ZipInfo entries. https://bugs.python.org/issue43547

Attributes:

CRC

comment

compress_size

compress_type

create_system

create_version

date_time

external_attr

extra

extract_version

file_size

filename

flag_bits

header_offset

internal_attr

orig_filename

reserved

volume

Methods

FileHeader([zip64])	Return the per-file header as a bytes object.
from_file(filename[, arcname, strict_timestamps])	Construct an appropriate ZipInfo for a file on the filesystem.
is_dir()	Return True if this archive member is a directory.

CRC

comment

compress_size

compress_type

create_system

create_version

date_time

external_attr

extra

extract_version

file_size

filename

flag_bits

header_offset

internal_attr

orig_filename

reserved

volume

class fdroidserver.common.ColorFormatter(msg)

Bases: Formatter

Methods

converter([seconds])	Convert seconds since the Epoch to a time tuple expressing local time.
format(record)	Format the specified record as text.
formatException(ei)	Format and return the specified exception information as a string.
formatStack(stack_info)	This method is provided as an extension point for specialized formatting of stack information.
formatTime(record[, datefmt])	Return the creation time of the specified LogRecord as formatted text.
usesTime()	Check if the format uses the creation time of the record.

formatMessage

format(record)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class fdroidserver.common.Encoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: JSONEncoder

Methods

default(obj)	Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).
encode(o)	Return a JSON string representation of a Python data structure.
iterencode(o[, _one_shot])	Encode the given object and yield each string representation as available.

default(obj)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

fdroidserver.common.FDroidPopen(commands, cwd=None, envs=None, output=True, stderr_to_stdout=True)

Run a command and capture the possibly huge output as a str.

Parameters:

commands

command and argument list like in subprocess.Popen

cwd

optionally specifies a working directory

envs

a optional dictionary of environment variables and their values

Returns:

A PopenResult.

fdroidserver.common.FDroidPopenBytes(commands, cwd=None, envs=None, output=True, stderr_to_stdout=True)

Run a command and capture the possibly huge output as bytes.

Parameters:

commands

command and argument list like in subprocess.Popen

cwd

optionally specifies a working directory

envs

a optional dictionary of environment variables and their values

Returns:

A PopenResult.

class fdroidserver.common.KnownApks

Bases: object

Permanent store of existing APKs with the date they were added.

This is currently the only way to permanently store the “updated” date of APKs.

Methods

recordapk(apkName[, default_date])

Record an APK (if it's new, otherwise does nothing).

recordapk(apkName, default_date=None)

Record an APK (if it’s new, otherwise does nothing).

Returns:

datetime

the date it was added as a datetime instance.

class fdroidserver.common.PopenResult(returncode=None, output=None)

Bases: object

fdroidserver.common.SdkToolsPopen(commands, cwd=None, output=True)

fdroidserver.common.ant_subprojects(root_dir)

fdroidserver.common.apk_extract_signatures(apkpath, outdir)

Extract a signature files from APK and puts them into target directory.

Parameters:

apkpath

location of the apk

outdir

older where the extracted signature files will be stored

References

• https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html

• https://source.android.com/security/apksigning/v2

• https://source.android.com/security/apksigning/v3

fdroidserver.common.apk_has_v1_signatures(apkfile)

Test whether an APK has v1 signature files.

fdroidserver.common.apk_implant_signatures(apkpath, outpath, manifest)

Implant a signature from metadata into an APK.

Note: this changes there supplied APK in place. So copy it if you need the original to be preserved.

Parameters:

apkpath

location of the unsigned apk

outpath

location of the output apk

References

• https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html

• https://source.android.com/security/apksigning/v2

• https://source.android.com/security/apksigning/v3

fdroidserver.common.apk_parse_release_filename(apkname)

Parse the name of an APK file according the F-Droids APK naming scheme.

WARNING: Returned values don’t necessarily represent the APKs actual properties, the are just paresed from the file name.

Returns:

Tuple

A triplet containing (appid, versionCode, signer), where appid should be the package name, versionCode should be the integer represion of the APKs version and signer should be the first 7 hex digists of the sha256 signing key fingerprint which was used to sign this APK.

fdroidserver.common.apk_signer_fingerprint(apk_path)

Get SHA-256 fingerprint string for the first signer from given APK.

Parameters:

apk_path

path to APK

Returns:

Standard SHA-256 signer fingerprint

fdroidserver.common.apk_signer_fingerprint_short(apk_path)

Get 7 hex digit SHA-256 fingerprint string for the first signer from given APK.

Parameters:

apk_path

path to APK

Returns:

first 7 chars of the standard SHA-256 signer fingerprint

fdroidserver.common.apk_strip_v1_signatures(signed_apk, strip_manifest=False)

Remove signatures from APK.

Parameters:

signed_apk

path to APK file.

strip_manifest

when set to True also the manifest file will be removed from the APK.

fdroidserver.common.app_matches_packagename(app, package)

fdroidserver.common.append_filename_to_mirrors(filename, mirrors)

Append the filename to all “url” entries in the mirrors dict.

fdroidserver.common.assert_config_keystore(config)

Check weather keystore is configured correctly and raise exception if not.

fdroidserver.common.auto_install_ndk(build)

Auto-install the NDK in the build, this assumes its in a buildserver guest VM.

Download, verify, and install the NDK version as specified via the “ndk:” field in the build entry. As it uncompresses the zipball, this forces the permissions to work for all users, since this might uncompress as root and then be used from a different user.

This needs to be able to install multiple versions of the NDK, since this is also used in CI builds, where multiple fdroid build –onserver calls can run in a single session. The production buildserver is reset between every build.

The default ANDROID_SDK_ROOT base dir of /opt/android-sdk is hard-coded in buildserver/Vagrantfile. The $ANDROID_HOME/ndk subdir is where Android Studio will install the NDK into versioned subdirs. https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41

Also, r10e and older cannot be handled via this mechanism because they are packaged differently.

fdroidserver.common.calculate_IPFS_cid(filename)

Calculate the IPFS CID of a file and add it to the index.

uses ipfs_cid package at https://packages.debian.org/sid/ipfs-cid Returns CIDv1 of a file as per IPFS recommendation

fdroidserver.common.calculate_math_string(expr)

fdroidserver.common.check_system_clock(dt_obj, path)

Check if system clock is updated based on provided date.

If an APK has files newer than the system time, suggest updating the system clock. This is useful for offline systems, used for signing, which do not have another source of clock sync info. It has to be more than 24 hours newer because ZIP/APK files do not store timezone info

fdroidserver.common.compare_apks(apk1, apk2, tmp_dir, log_dir=None)

Compare two apks.

Returns:

None if the APK content is the same (apart from the signing key),

otherwise a string describing what’s different, or what went wrong when

trying to do the comparison.

fdroidserver.common.config_type_check(path, data)

fdroidserver.common.deploy_build_log_with_rsync(appid, vercode, log_content)

Upload build log of one individual app build to an fdroid repository.

Parameters:

appid

package name for dientifying to which app this log belongs.

vercode

version of the app to which this build belongs.

log_content

Content of the log which is about to be posted. Should be either a string or bytes. (bytes will be decoded as ‘utf-8’)

fdroidserver.common.ensure_final_value(packageName, arsc, value)

Ensure incoming value is always the value, not the resid.

androguard will sometimes return the Android “resId” aka Resource ID instead of the actual value. This checks whether the value is actually a resId, then performs the Android Resource lookup as needed.

fdroidserver.common.fetch_real_name(app_dir, flavours)

Retrieve the package name. Returns the name, or None if not found.

fdroidserver.common.file_entry(filename, hash_value=None)

fdroidserver.common.fill_config_defaults(thisconfig)

Fill in the global config dict with relevant defaults.

For config values that have a path that can be expanded, e.g. an env var or a ~/, this will store the original value using “_orig” appended to the key name so that if the config gets written out, it will preserve the original, unexpanded string.

fdroidserver.common.find_apksigner(config)

Search for the best version apksigner and adds it to the config.

Returns the best version of apksigner following this algorithm:

• use config[‘apksigner’] if set

• try to find apksigner in path

• find apksigner in build-tools starting from newest installed going down to MINIMUM_APKSIGNER_BUILD_TOOLS_VERSION

Returns:

str

path to apksigner or None if no version is found

fdroidserver.common.find_command(command)

Find the full path of a command, or None if it can’t be found in the PATH.

fdroidserver.common.find_sdk_tools_cmd(cmd)

Find a working path to a tool from the Android SDK.

fdroidserver.common.force_exit(exitvalue=0)

Force exit when thread operations could block the exit.

The build command has to use some threading stuff to handle the timeout and locks. This seems to prevent the command from exiting, unless this hack is used.

fdroidserver.common.fsearch_g(/, string, pos=0, endpos=sys.maxsize)

Scan through string looking for a match, and return a corresponding match object instance.

Return None if no position in the string matches.

fdroidserver.common.genkeystore(localconfig)

Generate a new key with password provided in localconfig and add it to new keystore.

Parameters:

localconfig

Returns:

hexed public key, public key fingerprint

fdroidserver.common.genpassword()

Generate a random password for when generating keys.

fdroidserver.common.get_androguard_APK(apkfile, skip_analysis=False)

fdroidserver.common.get_android_tools_version_log()

Get a list of the versions of all installed Android SDK/NDK components.

fdroidserver.common.get_android_tools_versions()

Get a list of the versions of all installed Android SDK/NDK components.

fdroidserver.common.get_apk_id(apkfile)

Extract identification information from APK.

Androguard is preferred since it is more reliable and a lot faster. Occasionally, when androguard can’t get the info from the APK, aapt still can. So aapt is also used as the final fallback method.

Parameters:

apkfile

path to an APK file.

Returns:

appid

version code

version name

fdroidserver.common.get_apk_id_aapt(apkfile)

Read (appid, versionCode, versionName) from an APK.

fdroidserver.common.get_apk_id_androguard(apkfile)

Read (appid, versionCode, versionName) from an APK.

This first tries to do quick binary XML parsing to just get the values that are needed. It will fallback to full androguard parsing, which is slow, if it can’t find the versionName value or versionName is set to a Android String Resource (e.g. an integer hex value that starts with @).

This function is part of androguard as get_apkid(), so this vendored and modified to return versionCode as an integer.

fdroidserver.common.get_apksigner_smartcardoptions(smartcardoptions)

fdroidserver.common.get_app_display_name(app)

Get a human readable name for the app for logging and sorting.

When trying to find a localized name, this first tries en-US since that his the historical language used for sorting.

fdroidserver.common.get_build_dir(app)

Get the dir that this app will be built in.

fdroidserver.common.get_cachedir()

fdroidserver.common.get_cert_fingerprint(pubkey)

Generate a certificate fingerprint the same way keytool does it (but with slightly different formatting).

fdroidserver.common.get_certificate(signature_block_file, signature_file=None)

Extract a single DER certificate from JAR Signature’s “Signature Block File”.

If there is more than one signer certificate, this exits with an error, unless the signature_file is provided. If that is set, it will return the certificate that matches the Signature File, for example, if there is a certificate chain, like TLS does. In the fdroidserver use cases, there should always be a single signer. But rarely, some APKs include certificate chains.

This could be replaced by androguard’s APK.get_certificate_der() provided the cert chain fix was merged there. Maybe in 4.1.2? https://github.com/androguard/androguard/pull/1038

https://docs.oracle.com/en/java/javase/21/docs/specs/man/jarsigner.html#the-signed-jar-file

Parameters:

signature_block_file

Bytes representing the PKCS#7 signer certificate and signature, as read directly out of the JAR/APK, e.g. CERT.RSA.

signature_file

Bytes representing the manifest signed by the Signature Block File, e.g. CERT.SF. If this is not given, the assumption is there will be only a single certificate in signature_block_file, otherwise it is an error.

Returns:

A binary representation of the certificate’s public key,

or None in case of error

fdroidserver.common.get_config()

Get the initalized, singleton config instance.

config and options are intertwined in read_config(), so they have to be here too. In the current ugly state of things, there are multiple potential instances of config and options in use:

• global

• module-level in the subcommand module (e.g. fdroidserver/build.py)

• module-level in fdroidserver.common

There are some insane parts of the code that are probably referring to multiple instances of these at different points. This can be super confusing and maddening.

The current intermediate refactoring step is to move all subcommands to always get/set config and options via this function so that there is no longer a distinction between the global and module-level instances. Then there can be only one module-level instance in fdroidserver.common.

fdroidserver.common.get_default_cachedir()

Get a cachedir, using platformdirs for cross-platform, but works without.

Once platformdirs is installed everywhere, this function can be removed.

fdroidserver.common.get_dir_size(path_or_str)

Get the total size of all files in the given directory.

fdroidserver.common.get_effective_target_sdk_version(apk)

Wrap the androguard function to always return an integer.

Parameters:

apk

androguard APK object

Returns:

targetSdkVersion: int

fdroidserver.common.get_examples_dir()

Return the dir where the fdroidserver example files are available.

fdroidserver.common.get_extension(filename)

Get name and extension of filename, with extension always lower case.

fdroidserver.common.get_file_extension(filename)

Get the normalized file extension, can be blank string but never None.

fdroidserver.common.get_first_signer_certificate(apkpath)

Get the first signing certificate from the APK, DER-encoded.

JAR and APK Signatures allow for multiple signers, though it is rarely used, and this is poorly documented. So this method only fetches the first certificate, and errors out if there are more.

Starting with targetSdkVersion 30, APK v2 Signatures are required. https://developer.android.com/about/versions/11/behavior-changes-11#minimum-signature-scheme

When a APK v2+ signature is present, the JAR signature is not verified. The verifier parses the signers from the v2+ signature and does not seem to look at the JAR signature. https://source.android.com/docs/security/features/apksigning/v2#apk-signature-scheme-v2-block https://android.googlesource.com/platform/tools/apksig/+/refs/tags/android-13.0.0_r3/src/main/java/com/android/apksig/ApkVerifier.java#270

apksigner checks that the signers from all the APK signatures match: https://android.googlesource.com/platform/tools/apksig/+/refs/tags/android-13.0.0_r3/src/main/java/com/android/apksig/ApkVerifier.java#383

apksigner verifies each signer’s signature block file .(RSA|DSA|EC) against the corresponding signature file .SF https://android.googlesource.com/platform/tools/apksig/+/refs/tags/android-13.0.0_r3/src/main/java/com/android/apksig/internal/apk/v1/V1SchemeVerifier.java#280

NoOverwriteDict is a workaround for: https://github.com/androguard/androguard/issues/1030

Lots more discusion here: https://gitlab.com/fdroid/fdroidserver/-/issues/1128

fdroidserver.common.get_head_commit_id(git_repo)

Get git commit ID for HEAD as a str.

fdroidserver.common.get_jar_signer_certificate(pkcs7obj: ContentInfo, signature_file: bytes)

Return the one certificate in a chain that actually signed the manifest.

PKCS#7-signed data can include certificate chains for use cases where an Certificate Authority (CA) is used. Android does not validate the certificate chain on APK signatures, so neither does this. https://android.googlesource.com/platform/tools/apksig/+/refs/tags/android-13.0.0_r3/src/main/java/com/android/apksig/internal/apk/v1/V1SchemeVerifier.java#512

Some useful fodder for understanding all this: https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html https://technotes.shemyak.com/posts/jar-signature-block-file-format/ https://docs.oracle.com/en/java/javase/21/docs/specs/man/jarsigner.html#the-signed-jar-file https://qistoph.blogspot.com/2012/01/manual-verify-pkcs7-signed-data-with.html

fdroidserver.common.get_library_references(root_dir)

fdroidserver.common.get_local_metadata_files()

Get any metadata files local to an app’s source repo.

This tries to ignore anything that does not count as app metdata, including emacs cruft ending in ~

fdroidserver.common.get_metadata_files(vercodes)

Build a list of metadata files and raise an exception for invalid appids.

Parameters:

vercodes

version codes as returned by read_pkg_args()

Returns:

List

a list of corresponding metadata/*.yml files

fdroidserver.common.get_min_sdk_version(apk)

Wrap the androguard function to always return an integer.

Fall back to 1 if we can’t get a valid minsdk version.

Parameters:

apk

androguard APK object

Returns:

minSdkVersion: int

fdroidserver.common.get_mirrors(url, filename=None)

Get list of dict entries for mirrors, appending filename if provided.

fdroidserver.common.get_native_code(apkfile)

Aapt checks if there are architecture folders under the lib/ folder.

We are simulating the same behaviour.

fdroidserver.common.get_ndk_version(ndk_path)

Get the version info from the metadata in the NDK package.

Since r11, the info is nice and easy to find in sources.properties. Before, there was a kludgey format in RELEASE.txt. This is only needed for r10e.

fdroidserver.common.get_options()

Return options as set up by parse_args().

This provides an easy way to get the global instance without having to think about very confusing import and submodule visibility. The code should be probably refactored so it does not need this. If each individual option value was always passed to functions as args, for example.

https://docs.python.org/3/reference/import.html#submodules

fdroidserver.common.get_per_app_repos()

Per-app repos are dirs named with the packageName of a single app.

fdroidserver.common.get_release_filename(app, build, extension=None)

fdroidserver.common.get_toolsversion_logname(app, build)

fdroidserver.common.getpaths(build_dir, globpaths)

Extend via globbing the paths from a field and return them as a set.

fdroidserver.common.getpaths_map(build_dir, globpaths)

Extend via globbing the paths from a field and return them as a map from original path to resulting paths.

fdroidserver.common.getsrclib(spec, srclib_dir, basepath=False, raw=False, prepare=True, preponly=False, refresh=True, build=None)

Get the specified source library.

Return the path to it. Normally this is the path to be used when referencing it, which may be a subdirectory of the actual project. If you want the base directory of the project, pass ‘basepath=True’.

spec and srclib_dir are both strings, not pathlib.Path.

fdroidserver.common.getsrclibvcs(name)

fdroidserver.common.getsrcname(app, build)

fdroidserver.common.getvcs(vcstype, remote, local)

Return a vcs instance based on the arguments.

remote and local can be either a string or a pathlib.Path

fdroidserver.common.is_debuggable_or_testOnly(apkfile)

Return True if the given file is an APK and is debuggable or testOnly.

These two settings should never be enabled in release builds. This parses <application android:debuggable=”” android:testOnly=””> from the APK and nothing else to run fast, since it is run on every APK as part of update.

Parameters:

apkfile

full path to the APK to check

fdroidserver.common.is_repo_file(filename, for_gpg_signing=False)

Whether the file in a repo is a build product to be delivered to users.

fdroidserver.common.is_strict_application_id(name)

Check whether name is a valid Android Application ID.

The Android ApplicationID is basically a Java Package Name, but with more restrictive naming rules:

• It must have at least two segments (one or more dots).

• Each segment must start with a letter.

• All characters must be alphanumeric or an underscore [a-zA-Z0-9_].

References

https://developer.android.com/studio/build/application-id

fdroidserver.common.is_valid_package_name(name)

Check whether name is a valid fdroid package name.

APKs and manually defined package names must use a valid Java Package Name. Automatically generated package names for non-APK files use the SHA-256 sum.

fdroidserver.common.load_localized_config(name, repodir)

Load localized config files and put them into internal dict format.

This will maintain the order as came from the data files, e.g YAML. The locale comes from unsorted paths on the filesystem, so that is separately sorted.

fdroidserver.common.load_stats_fdroid_signing_key_fingerprints()

Load signing-key fingerprints stored in file generated by fdroid publish.

Returns:

dict

containing the signing-key fingerprints.

fdroidserver.common.local_rsync(options, from_paths: List[str], todir: str)

Rsync method for local to local copying of things.

This is an rsync wrapper with all the settings for safe use within the various fdroidserver use cases. This uses stricter rsync checking on all files since people using offline mode are already prioritizing security above ease and speed.

fdroidserver.common.manifest_paths(app_dir, flavours)

Return list of existing files that will be used to find the highest vercode.

fdroidserver.common.metadata_find_developer_signature(appid, vercode=None)

Try to find the developer signature for given appid.

This picks the first signature file found in metadata an returns its signature.

Returns:

sha256 signing key fingerprint of the developer signing key.

None in case no signature can not be found.

fdroidserver.common.metadata_find_developer_signing_files(appid, vercode)

Get developer signature files for specified app from metadata.

Returns:

List

of 4-tuples for each signing key with following paths: (signature_file, signature_block_file, manifest, v2_files), where v2_files is either a (apk_signing_block_offset_file, apk_signing_block_file) pair or None

fdroidserver.common.metadata_find_signing_files(appid, vercode)

Get a list of signed manifests and signatures.

Parameters:

appid

app id string

vercode

app version code

Returns:

List

of 4-tuples for each signing key with following paths: (signature_file, signature_block_file, manifest, v2_files), where v2_files is either a (apk_signing_block_offset_file, apk_signing_block_file) pair or None

References

• https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html

• https://source.android.com/security/apksigning/v2

• https://source.android.com/security/apksigning/v3

fdroidserver.common.metadata_get_sigdir(appid, vercode=None)

Get signature directory for app.

fdroidserver.common.natural_key(s)

fdroidserver.common.parse_androidmanifests(paths, app)

Extract some information from the AndroidManifest.xml at the given path.

Returns (version, vercode, package), any or all of which might be None. All values returned are strings.

Android Studio recommends “you use UTF-8 encoding whenever possible”, so this code assumes the files use UTF-8. https://sites.google.com/a/android.com/tools/knownissues/encoding

fdroidserver.common.parse_args(parser)

Call parser.parse_args(), store result in module-level variable and return it.

This is needed to set up the copy of the options instance in the fdroidserver.common module. A subcommand only needs to call this if it uses functions from fdroidserver.common that expect the “options” variable to be initialized.

fdroidserver.common.parse_human_readable_size(size)

fdroidserver.common.parse_mirrors_config(mirrors)

Mirrors can be specified as a string, list of strings, or dictionary map.

fdroidserver.common.parse_srclib_spec(spec)

fdroidserver.common.parse_xml(path)

fdroidserver.common.place_srclib(root_dir, number, libpath)

fdroidserver.common.prepare_source(vcs, app, build, build_dir, srclib_dir, extlib_dir, onserver=False, refresh=True)

Prepare the source code for a particular build.

Parameters:

vcs

the appropriate vcs object for the application

app

the application details from the metadata

build

the build details from the metadata

build_dir

the path to the build directory, usually ‘build/app.id’

srclib_dir

the path to the source libraries directory, usually ‘build/srclib’

extlib_dir

the path to the external libraries directory, usually ‘build/extlib’

Returns:

root

is the root directory, which may be the same as ‘build_dir’ or may be a subdirectory of it.

srclibpaths

is information on the srclibs being used

fdroidserver.common.psearch_g(/, string, pos=0, endpos=sys.maxsize)

Scan through string looking for a match, and return a corresponding match object instance.

Return None if no position in the string matches.

fdroidserver.common.publishednameinfo(filename)

fdroidserver.common.read_app_args(appid_versionCode_pairs, allow_version_codes=False, sort_by_time=False)

Build a list of App instances for processing.

On top of what read_pkg_args does, this returns the whole app metadata, but limiting the builds list to the builds matching the appid_versionCode_pairs and vercodes specified. If no appid_versionCode_pairs are specified, then all App and Build instances are returned.

fdroidserver.common.read_config()

Read the repository config.

The config is read from config_file, which is in the current directory when any of the repo management commands are used. If there is a local metadata file in the git repo, then the config is not required, just use defaults.

config.yml is the preferred form because no code is executed when reading it. config.py is deprecated and supported for backwards compatibility.

config.yml requires ASCII or UTF-8 encoding because this code does not auto-detect the file’s encoding. That is left up to the YAML library. YAML allows ASCII, UTF-8, UTF-16, and UTF-32 encodings. Since it is a good idea to manage config.yml (WITHOUT PASSWORDS!) in git, it makes sense to use a globally standard encoding.

fdroidserver.common.read_pkg_args(appid_versionCode_pairs, allow_version_codes=False)

No summary.

Parameters:

appids

arguments in the form of multiple appid:[versionCode] strings

Returns:

a dictionary with the set of vercodes specified for each package

fdroidserver.common.regsub_file(pattern, repl, path)

fdroidserver.common.remove_debuggable_flags(root_dir)

fdroidserver.common.remove_signing_keys(build_dir)

fdroidserver.common.replace_build_vars(cmd, build)

fdroidserver.common.replace_config_vars(cmd, build)

fdroidserver.common.retrieve_string(app_dir, string, xmlfiles=None)

fdroidserver.common.retrieve_string_singleline(app_dir, string, xmlfiles=None)

fdroidserver.common.rsync_status_file_to_repo(path, repo_subdir=None)

Copy a build log or status JSON to the repo using rsync.

fdroidserver.common.run_yamllint(path, indent=0)

fdroidserver.common.set_FDroidPopen_env(build=None)

Set up the environment variables for the build environment.

There is only a weak standard, the variables used by gradle, so also set up the most commonly used environment variables for SDK and NDK. Also, if there is no locale set, this will set the locale (e.g. LANG) to en_US.UTF-8.

fdroidserver.common.set_command_in_config(command)

Try to find specified command in the path, if it hasn’t been manually set in config.yml.

If found, it is added to the config dict. The return value says whether the command is available.

fdroidserver.common.set_console_logging(verbose=False, color=False)

Globally set logging to output nicely to the console.

fdroidserver.common.setup_global_opts(parser)

fdroidserver.common.setup_status_output(start_timestamp)

Create the common output dictionary for public status updates.

fdroidserver.common.setup_vcs(app)

Checkout code from VCS and return instance of vcs and the build dir.

fdroidserver.common.sha256base64(filename)

Calculate the sha256 of the given file as URL-safe base64.

fdroidserver.common.sha256sum(filename)

Calculate the sha256 of the given file.

fdroidserver.common.sign_apk(unsigned_path, signed_path, keyalias)

Sign an unsigned APK, then save to a new file, deleting the unsigned.

NONE is a Java keyword used to configure smartcards as the keystore. Otherwise, the keystore is a local file. https://docs.oracle.com/javase/7/docs/technotes/guides/security/p11guide.html#KeyToolJarSigner

When using smartcards, apksigner does not use the same options has Java/keytool/jarsigner (-providerName, -providerClass, -providerArg, -storetype). apksigner documents the options as –ks-provider-class and –ks-provider-arg. Those seem to be accepted but fail when actually making a signature with weird internal exceptions. We use the options that actually work. From: https://geoffreymetais.github.io/code/key-signing/#scripting

fdroidserver.common.signer_fingerprint(cert_encoded)

Return SHA-256 signer fingerprint for PKCS#7 DER-encoded signature.

Parameters:

Contents of an APK signature.

Returns:

Standard SHA-256 signer fingerprint.

fdroidserver.common.signer_fingerprint_short(cert_encoded)

Obtain shortened sha256 signing-key fingerprint for pkcs7 DER certficate.

Extracts the first 7 hexadecimal digits of sha256 signing-key fingerprint for a given pkcs7 signature.

Parameters:

cert_encoded

Contents of an APK signing certificate.

Returns:

shortened signing-key fingerprint.

fdroidserver.common.string_is_integer(string)

fdroidserver.common.test_aapt_version(aapt)

Check whether the version of aapt is new enough.

fdroidserver.common.test_sdk_exists(thisconfig)

fdroidserver.common.unescape_string(string)

class fdroidserver.common.vcs(remote, local)

Bases: object

Methods

deinitsubmodules()
getref([revname])	Get current commit reference (hash, revision, etc).
getsrclib()	Return the srclib (name, path) used in setting up the current revision, or None.
gettags()
gotorevision(rev[, refresh])	Take the local repository to a clean version of the given revision.
gotorevisionx(rev)	No summary.
initsubmodules()
latesttags()	Get a list of all the known tags, sorted from newest to oldest.

clientversion
clientversioncmd
repotype

clientversion()

clientversioncmd()

deinitsubmodules()

getref(revname=None)

Get current commit reference (hash, revision, etc).

getsrclib()

Return the srclib (name, path) used in setting up the current revision, or None.

gettags()

gotorevision(rev, refresh=True)

Take the local repository to a clean version of the given revision.

Take the local repository to a clean version of the given revision, which is specificed in the VCS’s native format. Beforehand, the repository can be dirty, or even non-existent. If the repository does already exist locally, it will be updated from the origin, but only once in the lifetime of the vcs object. None is acceptable for ‘rev’ if you know you are cloning a clean copy of the repo - otherwise it must specify a valid revision.

gotorevisionx(rev)

No summary.

Derived classes need to implement this.

It’s called once basic checking has been performed.

initsubmodules()

latesttags()

Get a list of all the known tags, sorted from newest to oldest.

repotype()

class fdroidserver.common.vcs_bzr(remote, local)

Bases: vcs

Methods

bzr(args[, envs, cwd, output])	Prevent bzr from ever using SSH to avoid security vulns.
deinitsubmodules()
getref([revname])	Get current commit reference (hash, revision, etc).
getsrclib()	Return the srclib (name, path) used in setting up the current revision, or None.
gettags()
gotorevision(rev[, refresh])	Take the local repository to a clean version of the given revision.
gotorevisionx(rev)	No summary.
initsubmodules()
latesttags()	Get a list of all the known tags, sorted from newest to oldest.

clientversion
clientversioncmd
repotype

bzr(args, envs={}, cwd=None, output=True)

Prevent bzr from ever using SSH to avoid security vulns.

clientversioncmd()

gotorevisionx(rev)

No summary.

Derived classes need to implement this.

It’s called once basic checking has been performed.

repotype()

class fdroidserver.common.vcs_git(remote, local)

Bases: vcs

Methods

checkrepo()	No summary.
getref([revname])	Get current commit reference (hash, revision, etc).
getsrclib()	Return the srclib (name, path) used in setting up the current revision, or None.
gettags()
git(args[, envs, cwd, output])	Prevent git fetch/clone/submodule from hanging at the username/password prompt.
gotorevision(rev[, refresh])	Take the local repository to a clean version of the given revision.
gotorevisionx(rev)	No summary.
latesttags()	Return a list of latest tags.

clientversion
clientversioncmd
deinitsubmodules
initsubmodules
repotype

checkrepo()

No summary.

If the local directory exists, but is somehow not a git repository, git will traverse up the directory tree until it finds one that is (i.e. fdroidserver) and then we’ll proceed to destroy it! This is called as a safety check.

clientversioncmd()

deinitsubmodules()

getref(revname='HEAD')

Get current commit reference (hash, revision, etc).

git(args, envs={}, cwd=None, output=True)

Prevent git fetch/clone/submodule from hanging at the username/password prompt.

While fetch/pull/clone respect the command line option flags, it seems that submodule commands do not. They do seem to follow whatever is in env vars, if the version of git is new enough. So we just throw the kitchen sink at it to see what sticks.

Also, because of CVE-2017-1000117, block all SSH URLs.

gotorevisionx(rev)

No summary.

Derived classes need to implement this.

It’s called once basic checking has been performed.

initsubmodules()

latesttags()

Return a list of latest tags.

repotype()

class fdroidserver.common.vcs_gitsvn(remote, local)

Bases: vcs

Methods

checkrepo()	No summary.
deinitsubmodules()
getref([revname])	Get current commit reference (hash, revision, etc).
getsrclib()	Return the srclib (name, path) used in setting up the current revision, or None.
gettags()
git(args[, envs, cwd, output])	Prevent git fetch/clone/submodule from hanging at the username/password prompt.
gotorevision(rev[, refresh])	Take the local repository to a clean version of the given revision.
gotorevisionx(rev)	No summary.
initsubmodules()
latesttags()	Get a list of all the known tags, sorted from newest to oldest.

clientversion
clientversioncmd
repotype

checkrepo()

No summary.

If the local directory exists, but is somehow not a git repository, git will traverse up the directory tree until it finds one that is (i.e. fdroidserver) and then we’ll proceed to destory it! This is called as a safety check.

clientversioncmd()

getref(revname='HEAD')

Get current commit reference (hash, revision, etc).

git(args, envs={}, cwd=None, output=True)

Prevent git fetch/clone/submodule from hanging at the username/password prompt.

AskPass is set to /bin/true to let the process try to connect without a username/password.

The SSH command is set to /bin/false to block all SSH URLs (supported in git >= 2.3). This protects against CVE-2017-1000117.

gotorevisionx(rev)

No summary.

Derived classes need to implement this.

It’s called once basic checking has been performed.

repotype()

class fdroidserver.common.vcs_hg(remote, local)

Bases: vcs

Methods

deinitsubmodules()
getref([revname])	Get current commit reference (hash, revision, etc).
getsrclib()	Return the srclib (name, path) used in setting up the current revision, or None.
gettags()
gotorevision(rev[, refresh])	Take the local repository to a clean version of the given revision.
gotorevisionx(rev)	No summary.
initsubmodules()
latesttags()	Get a list of all the known tags, sorted from newest to oldest.

clientversion
clientversioncmd
repotype

clientversioncmd()

gotorevisionx(rev)

No summary.

Derived classes need to implement this.

It’s called once basic checking has been performed.

repotype()

fdroidserver.common.vcsearch_g(/, string, pos=0, endpos=sys.maxsize)

Scan through string looking for a match, and return a corresponding match object instance.

Return None if no position in the string matches.

fdroidserver.common.verify_apk_signature(apk, min_sdk_version=None)

Verify the signature on an APK.

Try to use apksigner whenever possible since jarsigner is very shitty: unsigned APKs pass as “verified”! Warning, this does not work on JARs with apksigner >= 0.7 (build-tools 26.0.1)

Returns:

Boolean

whether the APK was verified

fdroidserver.common.verify_apks(signed_apk, unsigned_apk, tmp_dir, v1_only=None)

Verify that two apks are the same.

One of the inputs is signed, the other is unsigned. The signature metadata is transferred from the signed to the unsigned apk, and then apksigner is used to verify that the signature from the signed APK is also valid for the unsigned one. If the APK given as unsigned actually does have a signature, it will be stripped out and ignored.

Parameters:

signed_apk

Path to a signed APK file

unsigned_apk

Path to an unsigned APK file expected to match it

tmp_dir

Path to directory for temporary files

v1_only

True for v1-only signatures, False for v1 and v2 signatures, or None for autodetection

Returns:

None if the verification is successful, otherwise a string describing what went wrong.

fdroidserver.common.verify_deprecated_jar_signature(jar)

Verify the signature of a given JAR file, allowing deprecated algorithms.

index.jar (v0) and index-v1.jar are both signed by MD5/SHA1 by definition, so this method provides a way to verify those. Also, apksigner has different deprecation rules than jarsigner, so this is our current hack to try to represent the apksigner rules when executing jarsigner.

jarsigner is very shitty: unsigned JARs pass as “verified”! So this has to turn on -strict then check for result 4, since this does not expect the signature to be from a CA-signed certificate.

Also used to verify the signature on an archived APK, supporting deprecated algorithms.

F-Droid aims to keep every single binary that it ever published. Therefore, it needs to be able to verify APK signatures that include deprecated/removed algorithms. For example, jarsigner treats an MD5 signature as unsigned.

jarsigner passes unsigned APKs as “verified”! So this has to turn on -strict then check for result 4.

Just to be safe, this never reuses the file, and locks down the file permissions while in use. That should prevent a bad actor from changing the settings during operation.

Raises:

VerificationException

If the JAR’s signature could not be verified.

fdroidserver.common.verify_jar_signature(jar)

Verify the signature of a given JAR file.

jarsigner is very shitty: unsigned JARs pass as “verified”! So this has to turn on -strict then check for result 4, since this does not expect the signature to be from a CA-signed certificate.

Raises:

VerificationException

If the JAR’s signature could not be verified.

fdroidserver.common.version_code_string_to_int(vercode)

Convert an version code string of any base into an int.

fdroidserver.common.vnsearch_g(/, string, pos=0, endpos=sys.maxsize)

Scan through string looking for a match, and return a corresponding match object instance.

Return None if no position in the string matches.

fdroidserver.common.vnssearch_g(/, string, pos=0, endpos=sys.maxsize)

Scan through string looking for a match, and return a corresponding match object instance.

Return None if no position in the string matches.

fdroidserver.common.write_running_status_json(output)

fdroidserver.common.write_status_json(output, pretty=False, name=None)

Write status out as JSON, and rsync it to the repo server.

fdroidserver.common.write_to_config(thisconfig, key, value=None, config_file=None)

Write a key/value to the local config.yml or config.py.

NOTE: only supports writing string variables.

Parameters:

thisconfig

config dictionary

key

variable name in config to be overwritten/added

value

optional value to be written, instead of fetched from ‘thisconfig’ dictionary.