:py:mod:`oioioi.sinolpack.package` ================================== .. py:module:: oioioi.sinolpack.package Module Contents --------------- Classes ~~~~~~~ .. autoapisummary:: oioioi.sinolpack.package.SinolPackage oioioi.sinolpack.package.SinolPackageCreator oioioi.sinolpack.package.SinolPackageBackend Functions ~~~~~~~~~ .. autoapisummary:: oioioi.sinolpack.package._stringify_keys oioioi.sinolpack.package._determine_encoding oioioi.sinolpack.package._decode oioioi.sinolpack.package._make_filename_in_job_dir oioioi.sinolpack.package._remove_from_zip Attributes ~~~~~~~~~~ .. autoapisummary:: oioioi.sinolpack.package.logger oioioi.sinolpack.package.DEFAULT_TIME_LIMIT oioioi.sinolpack.package.DEFAULT_MEMORY_LIMIT oioioi.sinolpack.package.TASK_PRIORITY oioioi.sinolpack.package.C_EXTRA_ARGS oioioi.sinolpack.package.PAS_EXTRA_ARGS .. py:data:: logger .. py:data:: DEFAULT_TIME_LIMIT :annotation: = 10000 .. py:data:: DEFAULT_MEMORY_LIMIT :annotation: = 66000 .. py:data:: TASK_PRIORITY :annotation: = 500 .. py:data:: C_EXTRA_ARGS :annotation: = ['-Wall', '-Wno-unused-result', '-Werror'] .. py:data:: PAS_EXTRA_ARGS :annotation: = ['-Ci', '-Cr', '-Co', '-gl'] .. py:function:: _stringify_keys(dictionary) .. py:function:: _determine_encoding(title, file) .. py:function:: _decode(title, file) .. py:function:: _make_filename_in_job_dir(env, base_name) .. py:function:: _remove_from_zip(zipfname, *filenames) Removes files from zip file by creating new zip file with all the files except the files to remove. Then the old file is removed. It has to be done like this because zipfile module doesn't implement function to delete file. .. py:class:: SinolPackage(path, original_filename=None) Bases: :py:obj:`object` .. py:attribute:: controller_name :annotation: = oioioi.sinolpack.controllers.SinolProblemController .. py:attribute:: package_backend_name :annotation: = oioioi.sinolpack.package.SinolPackageBackend .. py:method:: identify() .. py:method:: get_short_name() .. py:method:: _find_main_dir() Looks for the directory which contains at least the in/ and out/ subdirectories. Only one such directory should be found. Otherwise None is returned. .. py:method:: _save_to_field(field, file) .. py:method:: _find_and_compile(suffix, command=None, cwd=None, log_on_failure=True, out_name=None) .. py:method:: _compile_using_make(command, cwd, suffix) .. py:method:: _compile_matching_extension(command, out_name, suffix) .. py:method:: _compile(filename, prog_name, ext, out_name=None) .. py:method:: _override_compiler(prefix, lang, compilation_job) .. py:method:: _run_compilation_job(ext, ft_source_name, out_name) .. py:method:: _ensure_compilation_success(filename, new_env) .. py:method:: _make_ins(re_string) .. py:method:: unpack(env, package) .. py:method:: _create_problem_or_reuse_if_exists(existing_problem) .. py:method:: _ensure_short_name_equality_with(existing_problem) .. py:method:: _create_problem_instance() .. py:method:: _extract_and_process_package() .. py:method:: _describe_processing_error() .. py:method:: _process_package() .. py:method:: _process_config_yml() Parses the ``config.yml`` file from the package. Extracted information is then saved for later use. .. py:method:: _detect_full_name() Sets the problem's full name from the ``config.yml`` (key ``title``) or from the ``title`` tag in the LaTeX source file. The ``config.yml`` file takes precedence over the LaTeX source. Example of how the ``title`` tag may look like: itle{A problem} .. py:method:: _detect_full_name_translations() Creates problem's full name translations from the ``config.yml`` (keys matching the pattern ``title_[a-z]{2}``, where ``[a-z]{2}`` represents two-letter language code defined in ``settings.py``), if any such key is given. .. py:method:: _detect_library() Finds if the problem has a library defined in ``config.yml``. Tries to read a library name (filename library should be given during compilation) from the ``config.yml`` (key ``library``). If there is no such key, assumes that a library is not needed. .. py:method:: _process_extra_files() Looks for extra compilation files specified in ``config.yml``. .. py:method:: _find_and_save_files(files) Saves files in the database. :param files: List of expected files. :return: List of files that were not found. .. py:method:: _extract_makefiles() .. py:method:: _save_prog_dir() Creates an archive to store compiled programs. .. py:method:: _process_statements() Creates a problem statement from html or pdf source. If `USE_SINOLPACK_MAKEFILES` is set to True in the OIOIOI settings, the pdf file will be compiled from a LaTeX source. .. py:method:: _force_index_encoding(htmlzipfile) Ensures index.html file is utf-8 encoded, if cannot apply this encoding raise :class:`~oioioi.problems.package.ProblemPackageError`. .. py:method:: _compile_latex_docs(docdir) .. py:method:: _generate_tests(total_score_if_auto=100) Generates tests for the problem. First, time and memory limits are obtained from ``config.yml``. Then, the judge system attempts to obtain a generic memory limit from the problem statement. Next, test instances are created, using input files (``*.in``) provided with the package and `ingen` (the input file generator) where applicable (``ingen``-generated tests may replace ``*.in`` files contained in the package if their respective names coincide). If `USE_SINOLPACK_MAKEFILES` is set to True in the OIOIOI settings, output files (``*.out``) will also be generated at this point, based on the model solution's output (on condition that its source code is included within the package). The next step is veryfing whether the sum of time limits over all tests does not exceed the maximum defined by the OIOIOI installation's owner. If an input verifier is provided, it will then assert that all ``*.in`` files generated are valid, and abort the upload in case of failure. Here the ``*.out`` files will be generated if `USE_SINOLPACK_MAKEFILES` is set to False, based on the model solution's output (on condition that its source code is included within the package). In the end, it is asserted that all tests have been correctly constructed, non-created tests are removed from the database and test scores are assigned to tests and testgroups based on the configuration from ``config.yml`` or set to default value if not specified. .. py:method:: _detect_statement_memory_limit() Returns the memory limit in the problem statement, converted to KiB or ``None``. .. py:method:: _create_instances_for_tests() Iterate through available test inputs. :return: Triple (created tests instances, outs that have to be generated, score groups (determined by test names)) .. py:method:: _verify_time_limits(time_limit_sum) Checks whether the sum of test time limits does not exceed the allowed maximum. :raises: :class:`~oioioi.problems.package.ProblemPackageError` if sum of tests time limits exceeds the maximum defined in the `settings.py` file. .. py:method:: _verify_inputs(tests) Checks if ``inwer`` exits with code 0 on all tests. :raises: :class:`~oioioi.problems.package.ProblemPackageError` otherwise. .. py:method:: _generate_test_outputs(tests, outs_to_make) .. py:method:: _validate_tests(created_tests) Checks if all tests have output files and that all tests have been successfully created. :raises: :class:`~oioioi.problems.package.ProblemPackageError` .. py:method:: _delete_non_existing_tests(created_tests) .. py:method:: _process_test(test, order, names_re, indir, outdir, collected_ins, scored_groups, outs_to_make) Responsible for saving test in and out files, setting test limits, assigning test kinds and groups. :param test: Test name. :param order: Test number. :param names_re: Compiled regex to match test details from name. Should extract basename, test name, group number and test type. :param indir: Directory with tests inputs. :param outdir: Directory with tests outputs. :param collected_ins: List of inputs that were generated, not taken from archive as a file. :param scored_groups: Accumulator for score groups. :param outs_to_make: Accumulator for name of output files to be generated by model solution. :return: Test instance or None if name couldn't be matched. .. py:method:: _get_memory_limit(created, name, group) If we find the memory limit specified anywhere in the package: either in the ``config.yml`` or in the problem statement then we overwrite potential manual changes. (In the future we should disallow editing memory limits if they were taken from the package). The memory limit is more important the more specific it is. In particular, the global memory limit is less important than the memory limit for a test group, while the memory limit for particular test is the most imporant. :return: Memory limit found in config or statement, None otherwise. .. py:method:: _get_time_limit(created, name, group) If we find the time limit specified anywhere in in the ``config.yml`` then we overwrite potential manual changes. The time limit is more important the more specific it is. In particular, the global time limit is less important than the time limit for a test group, while the time limit for particular test is the most imporant. :return: Time limit found in config, None otherwise. .. py:method:: _make_outs(outs_to_make) Compiles the model solution and executes it in order to generate test outputs. :return: Result from workers. .. py:method:: _check_scores_from_config(scored_groups, config_scores) Called if ``config.yml`` specifies scores for any tests. Makes sure that all scored tests are present in ``config.yml`` and that nothing else is there. .. py:method:: _compute_scores_automatically(scored_groups, total_score) If there are no testscores specified ``config.yml``, all groups get equal score, except few last groups that are given +1 to compensate rounding error and match the total sum of ``total_score``. .. py:method:: _assign_scores(scored_groups, total_score_if_auto) Checks if there's a ``scores`` entry in config and sets scores according to that or assigns them automatically otherwise. .. py:method:: _process_language_override() Checks if there's a `override_limits` entry in config and for existing tests, add additional limits overrides. Time limits are validated the same way it's validated in default package. .. py:method:: _prepare_overrides(lang) Prepares overrides for specified language, initially setting to default limits. .. py:method:: _set_memory_limit_overrides(lang, rules) Sets memory limits overrides for specific language. .. py:method:: _set_time_limit_overrides(lang, rules) Sets time limits overrides for specific language. .. py:method:: _process_checkers() Compiles an output checker and saves its binary. .. py:method:: _find_checker_exec() .. py:method:: _process_model_solutions() Saves model solutions to the database. .. py:method:: _get_model_solutions_sources() :return: Sources as tuples (kind_of_solution, filename, full path to file). .. py:method:: _process_attachments() Removes previously added attachments for the problem, and saves new ones from the attachment directory. .. py:method:: _save_original_package() Save instance of package that would be reused by other instances of this problem. .. py:method:: _html_disallowed() .. py:class:: SinolPackageCreator(problem) Bases: :py:obj:`object` Responsible for packing SinolPackages. .. py:method:: pack() :returns: Archive from original package if such file exists, otherwise new archive with test, statements and model solutions. .. py:method:: _create_basic_archive() Produce the most basic output: tests, statements, model solutions. .. py:method:: _pack_statement() .. py:method:: _pack_django_file(django_file, arcname) Packs file represented by :class:~`oioioi.filetracker.fields.FileField` .. py:method:: _pack_tests() .. py:method:: _pack_model_solutions() .. py:class:: SinolPackageBackend Bases: :py:obj:`oioioi.problems.package.ProblemPackageBackend` Backend that use :class:`~oioioi.sinolpack.package.SinolPackage` to unpack and :class:`~oioioi.sinolpack.package.SinolPackageCreator` to pack sinol packages. .. py:attribute:: description .. py:attribute:: package_class .. py:method:: identify(path, original_filename=None) Checks if the backend is suitable for processing the specified problem package. :param path: a path to the processed problem package :param original_filename: the name of the package specified by the uploading user. Returns ``True`` if the backend can handle the specified problem package file. .. py:method:: get_short_name(path, original_filename=None) Returns the problem's short name. :param path: a path to the processed problem package :param original_filename: the name of the package specified by the uploading user. .. py:method:: unpack(env) Processes a package, creating a new :class:`~oioioi.problems.models.Problem` or updating an existing one. This function will be called either from :func:`~oioioi.problems.unpackmgr.unpackmgr_job` (Celery task) or from :func:`~oioioi.problems.package.simple_unpack` (e.g. when a problem is added from a command line). Used ``env`` keys: ``package_id``: an id of the :class:`~oioioi.problems.models.ProblemPackage` instance with the package file to unpack. Produced ``env`` keys: ``problem_id``: an id of the :class:`~oioioi.problems.models.Problem` instance representing the created or modified problem. .. py:method:: pack(problem) Creates a package from problem, returns a :class:`django.http.HttpResponse` instance. Should raise ``NotImplementedError`` if creating packages is not supported.