From: Not Zed Date: Thu, 25 Apr 2019 07:05:27 +0000 (+0930) Subject: modularised dez X-Git-Tag: dez-1.3~3 X-Git-Url: https://code.zedzone.au/cvs?a=commitdiff_plain;h=06f891165b000c3d0abb9310950fe7ad11c48c18;p=dez modularised dez --- 06f891165b000c3d0abb9310950fe7ad11c48c18 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..470b0ea --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +/nbproject/private/ +/bin/ +/build/ +/dist/ + diff --git a/COPYING.AGPL3 b/COPYING.AGPL3 new file mode 100644 index 0000000..dba13ed --- /dev/null +++ b/COPYING.AGPL3 @@ -0,0 +1,661 @@ + GNU AFFERO GENERAL PUBLIC LICENSE + Version 3, 19 November 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + + A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + + The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + + An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing under +this license. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU Affero General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Remote Network Interaction; Use with the GNU General Public License. + + Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your version +supports such interaction) an opportunity to receive the Corresponding +Source of your version by providing access to the Corresponding Source +from a network server at no charge, through some standard or customary +means of facilitating copying of software. This Corresponding Source +shall include the Corresponding Source for any work covered by version 3 +of the GNU General Public License that is incorporated pursuant to the +following paragraph. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU Affero General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for the +specific requirements. + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU AGPL, see +. diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..fd6dea6 --- /dev/null +++ b/Makefile @@ -0,0 +1,18 @@ + +dist_VERSION=1.2.99 +dist_NAME=dez +dist_EXTRA=README COPYING.AGPL3 junit.make \ + build.xml \ + nbproject/build-impl.xml \ + nbproject/genfiles.properties \ + nbproject/project.properties \ + nbproject/project.xml + +include config.make + +java_MODULES=notzed.dez notzed.dez.demo + +notzed.dez.demo_JDEPMOD = notzed.dez + +include java.make +include junit.make diff --git a/README b/README new file mode 100644 index 0000000..51c3ae3 --- /dev/null +++ b/README @@ -0,0 +1,106 @@ + +INTRODUCTION +------------ + +This is a basic binary delta generator for in-memory files. + +It is ostensibly based on the algorithm from Bentley & McIllroy's +paper ``Data Compression Using Long Common Strings''. Here however +the hashes may be performed on overlapping blocks, possibly at every +location. + +It includes an encoder and decoder for a proprietary (and still not +quite fixed) delta format 'DEZ1' but standardised formats such as +VCDIFF are possible by implementing a simple interface. + +The 'matcher' is also replaceable. + +COMPILING +--------- + +A GNU makefile is provided for GNU systems. A Java JDK supporting +modular Java is required. OpenJDK 11 was used for this version. + +$ make + +Will make the modular jar files. + +`bin/notzed.dez/notzed.dez.jar' + Main library +`bin/notzed.dez.demo/notzed.dez.demo.jar' + 'dez' tool and example classes. + +$ make check + +Will compile run the tests. They use junit4 from netbeans 11. The +make check target is still experimental, see junit.make. + +The netbeans 11 project files are also included. + +USING +----- + +See dez.java in the demo module, or the tests for an example of basic +api usage. + +dez.java implements a simple command line tool. + +Create a patch: + +$ java --module-path bin/modules -m notzed.dez.demo/au.notzed.dez.demo.dez \ + -c source target > patch + +The patch format is binary so redirecting the output is recommended. + +Decode a patch: + +$ java --module-path bin/modules -m notzed.dez.demo/au.notzed.dez.demo.dez \ + -d source patch > recovered + +Print detailed patch contents: + +$ java --module-path bin/modules -m notzed.dez.demo/au.notzed.dez.demo.dez \ + -p patch + +STATUS +------ + +As of 1.3 this could be considered pre-beta. The dez file format is +now likely fixed but still may change. At the very least it requires +a checksum for production use. + +There are now performance tunables to handle some nasty worst-case +behaviour due to implementing an exhaustive search. These trade off +delta size against cpu and memory use in the encoder. The decoder is +unchanged. + +It needs more testing. + +The makefile is still work in progress. + +LINKS +----- + +* DEZ + +LICENSE +------- + +The code is covered by the GNU Affero General Public License version 3 +(or later). See COPYING.AGPL3 for full details. + + Copyright (C) 2015,2019 Michael Zucchi + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as + published by the Free Software Foundation, either version 3 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public + License along with this program. If not, see + . diff --git a/build.xml b/build.xml new file mode 100644 index 0000000..9f9ab89 --- /dev/null +++ b/build.xml @@ -0,0 +1,65 @@ + + + + + + + + + + + Builds, tests, and runs the project notzed.dez. + + + diff --git a/config.make b/config.make new file mode 100644 index 0000000..657bb75 --- /dev/null +++ b/config.make @@ -0,0 +1,41 @@ + +TARGET ?= linux-amd64 + +JAVA_HOME ?= /usr/local/jdk-11.0.2 +JAVAFX_HOME ?= /usr/local/javafx-sdk-11.0.2 +FFMPEG_HOME ?= /opt/ffmpeg/4.0 + +# See also JAVACFLAGS --module-path +NATIVEZ_HOME=../nativez/bin/notzed.nativez/$(TARGET) + +JAVACFLAGS += --module-path $(JAVAFX_HOME)/lib:../nativez/bin/notzed.nativez +JAVACFLAGS += -source 11 + +JAVAC ?= javac +JAR ?= jar +JMOD ?= jmod + +# Linux options +# USE_SO_VERSION adds the major version to the library open name for ffmpeg libs on linux. +linux-amd64_CPPFLAGS = \ + -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux \ + -DUSE_SO_VERSION=1 +linux-amd64_CFLAGS = -fPIC -Os -Wall +linux-amd64_CC = cc +linux-amd64_LD = ld + +linux-amd64_SO = .so +linux-amd64_LIB = lib + +# Windows options +windows-amd64_CPPFLAGS = \ + -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux \ + -DHAVE_ALIGNED_MALLOC \ + -DWIN32 +windows-amd64_CFLAGS = -Os -Wall +windows-amd64_CC = x86_64-w64-mingw32-gcc +windows-amd64_LD = x86_64-w64-mingw32-ld +windows-amd64_LDFLAGS = -Wl,--subsystem,windows + +windows-amd64_SO = .dll +windows-amd64_LIB = diff --git a/java.make b/java.make new file mode 100644 index 0000000..7228203 --- /dev/null +++ b/java.make @@ -0,0 +1,314 @@ +# +# Copyright (C) 2019 Michael Zucchi +# +# This is the copyright for java.make +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# General purpose modular java makefile that supports native library +# compilation directly. Non-recrusve implementation. +# +# Uses metamake programming with some file conventions to implement +# auto-make-like features. + +# Define modules +# -------------- +# java_MODULES list of java modules to compile. The sources must +# exist in src//classes. Resource files are +# stored in src//classes. Source-code +# generators must exist in src//gen. Native +# libraries must exist in src//jni. + +# Global variables + +# JAVA_HOME location of jdk. +# JAVAC java compiler to use. Default is 'javac' on the path. +# JAVACFLAGS javac flags applied to all invocations. +# JAR jar command. +# JARFLAGS jar flags +# JMOD jmod command. +# JMODFLAGS jmod flags. + +# Module specific variables + +# _JDEPMOD Lists modules which this one depends on. + +# _JAVACFLAGS Extra module-specific flags for each command. +# _JARFLAGS +# _JMODFLAGS + +# _JAVA Java sources. If not set it is found from src//classes/(*.java) +# _RESOURCES .jar resources. If not set it is found from src//classes/(not *.java) +# _JAVA_GENERATED Java generated sources. These must be relative to the package name. + +# Variables for use in fragments + +# gen.make and jni.make can additionally make use of these variables + +# _gendir Location for files used in Java generation process (per project). +# _genjavadir Location where _JAVA_GENERATED .java files will be created (per project). +# _jnidir Location for jni generated files (per target). +# _objdir Location for c objects (per target). +# _incdir Location for output includes, .jmod staging. +# _libdir Location for output libraries, .jmod staging. May point to _bindir. +# _bindir Location for output commands, .jmod staging. + +# Define libraries +# ---------------- + +# Each module can define one or more native libraries. + +# These are compiled after the java sources have been compiled as that +# process also generates any native binding headers. + +# _JNI_LIBRARIES list of libraries to build. +# library names match System.loadLibrary(). + +# Global variables + +# _LDFLAGS +# _LDLIBS +# _CPPFLAGS +# _CFLAGS +# SO shared library suffix +# LIB shared library prefix + +# Utility functions +# +# $(call library-path,,) will resolve to the library file name. + +# Per library variables. + +# _SOURCES .c, .cc, .C - source files for library. Paths are relative to src//jni. +# _HEADERS header files for jmod +# _COMMANDS commands/bin/scripts for jmod + +# _LDFLAGS link flags +# _LIBADD extra objects to add to link line +# _LDLIBS link libraries +# _CPPFLAGS c pre-processor flags. "-Isrc//jni -Ibin/include/" is implicit. +# _CCFLAGS c compiler flags + +# _DEPENDENCIES A list of other objects on which this library depends before linking. + +# .c files have dependencies automatically generated + +# Targets +# ------- + +# make gen only generate java sources +# make clean rm -rf bin +# make dist create dist tar in bin/ +# make | make jar make all jars and jmods +# make bin make everything but jars and mods + +# Outputs +# ------- + +# All intermediate and output files are written to bin/ + +# This layout is enforced by javac +# bin/include// .h files from javac -h +# bin/modules// .class files from javac + +# This layout is convenient for netbeans +# bin/gen//gen/ .c, exe files for generator free use +# bin/gen//classes/ .java files from generator _JAVA_GENERATED + +# bin/status/ marker files for makefile + +# bin//.jar .jar modular + +# Native code + +# bin///lib .so librareies for jmod _LIBRARIES = libname +# bin///obj .o, .d files for library _SOURCES +# bin///include .h files for jmod _HEADERS +# bin///.jmod .jmod module + +# ###################################################################### + +E:= +S:=$(E) $(E) + +# All modules with native code +java_JMODS=$(foreach module,$(java_MODULES),$(if $(wildcard src/$(module)/jni/jni.make),$(module))) +# Only modules with no native code +java_JARS=$(foreach module,$(java_MODULES),$(if $(wildcard src/$(module)/jni/jni.make),,$(module))) +# Modules with generated java source +java_JGEN=$(foreach module,$(java_MODULES),$(if $(wildcard src/$(module)/gen/gen.make),$(module))) + +# Define some useful variables before including fragments +define java_variables= +$(1)_gendir:=bin/gen/$(1)/gen +$(1)_genjavadir:=bin/gen/$(1)/classes +$(1)_jnidir:=bin/$(1)/$(TARGET)/jni +$(1)_objdir:=bin/$(1)/$(TARGET)/obj +$(1)_incdir:=bin/$(1)/$(TARGET)/include +$(1)_libdir:=$$(if $$(filter windows-%,$(TARGET)),bin/$(1)/$(TARGET)/bin,bin/$(1)/$(TARGET)/lib) +$(1)_bindir:=bin/$(1)/$(TARGET)/bin +ifndef $(1)_JAVA +$(1)_JAVA := $$(shell find src/$(1)/classes -type f -name '*.java') +endif +ifndef $(1)_RESOURCES +$(1)_RESOURCES := $$(shell find src/$(1)/classes -type f \! -name '*.java') +endif +endef + +$(foreach module,$(java_MODULES),$(eval $(call java_variables,$(module)))) + +# ###################################################################### + +all: jar +bin: +gen: + +.PHONY: all clean jar bin gen +clean: + rm -rf bin + +include $(patsubst %,src/%/gen/gen.make,$(java_JGEN)) +include $(patsubst %,src/%/jni/jni.make,$(java_JMODS)) + +# ###################################################################### +# Java +# ###################################################################### + +define java_targets= +# Rules for module $(1) +$(1)_JAVA_generated = $$(addprefix $$($(1)_genjavadir)/,$$($(1)_JAVA_GENERATED)) + +bin/status/$(1).data: $$($(1)_RESOURCES) +bin/status/$(1).classes: $(patsubst %,bin/status/%.classes,$($(1)_JDEPMOD)) $$($(1)_JAVA) $$($(1)_JAVA_generated) +jar $(1): bin/$(1)/$(1).jar $(if $(wildcard src/$(1)/jni/jni.make),bin/$(1)/$(TARGET)/$(1).jmod) +bin: bin/status/$(1).classes bin/status/$(1).data +sources: bin/$(1)/$(1)-sources.zip +gen: $$($(1)_JAVA_generated) + +# Create modular jar +bin/$(1)/$(1).jar: bin/status/$(1).classes bin/status/$(1).data + @install -d $$(@D) + $(JAR) cf $$@ \ + $(JARFLAGS) $$($(1)_JARFLAGS) \ + -C bin/modules/$(1) . + +# Create a jmod +bin/$(1)/$(TARGET)/$(1).jmod: bin/status/$(1).classes bin/status/$(1).data + rm -f $$@ + @install -d $$(@D) + $$(JMOD) create \ + $$(JMODFLAGS) $$($(1)_JMODFLAGS) \ + --target-platform $(TARGET) \ + --class-path bin/modules/$(1) \ + $$(if $$(wildcard bin/$(1)/$(TARGET)/include),--header-files bin/$(1)/$(TARGET)/include) \ + $$(if $$(wildcard src/$(1)/legal),--legal-notices src/$(1)/legal) \ + $$(if $$(wildcard $$($(1)_bindir)),--cmds $$($(1)_bindir)) \ + $$(if $$(wildcard $$($(1)_libdir)),--libs $$($(1)_libdir)) \ + $$@ + +# Create an IDE source zip, paths have to match --module-source-path +bin/$(1)/$(1)-sources.zip: bin/status/$(1).classes + @install -d $$(@D) + jar -c -f $$@ -M \ + $$(patsubst src/$(1)/classes/%,-C src/$(1)/classes %,$$(filter src/$(1)/classes/%,$$($(1)_JAVA))) \ + $$(patsubst bin/gen/$(1)/classes/%,-C bin/gen/$(1)/classes %,$$(filter bin/gen/$(1)/classes/%,$$($(1)_JAVA))) + +endef + +#$(foreach module,$(java_MODULES),$(info $(call java_targets,$(module)))) +$(foreach module,$(java_MODULES),$(eval $(call java_targets,$(module)))) + +# ###################################################################### +# Global pattern rules + +# Stage resources +bin/status/%.data: + @install -d $(@D) + for data in $(patsubst src/$*/classes/%,"%",$($*_RESOURCES)) ; do \ + install -vDC "src/$*/classes/$$data" "bin/modules/$*/$$data" || exit 1 ; \ + done + touch $@ + +# Compile one module. This only updates (javac -h) headers if they changed. +bin/status/%.classes: + @install -d $(@D) + $(JAVAC) \ + --module-source-path "src/*/classes:bin/gen/*/classes" \ + $(JAVACFLAGS) $($*_JAVACFLAGS) \ + -h bin/inc \ + -d bin/modules \ + -m $* \ + $($*_JAVA) $($*_JAVA_generated) + if [ -d bin/inc/$* ] ; then \ + install -DC -t bin/include/$* bin/inc/$*/*.h ; \ + fi + touch $@ + +# ###################################################################### +# C stuff +# ###################################################################### + +SUFFIXES=.c .C .cc +SO=$($(TARGET)_SO) +LIB=$($(TARGET)_LIB) + +# functions to find cross-module stuff $(call library-path,modname,libname) +library-path=bin/$(1)/$(TARGET)/lib/$(LIB)$(2)$(SO) +library-dir=bin/$(1)/$(TARGET)/lib/ + +define jni_library= +# Rule for library $(2) in module $(1) +$(2)_OBJS = $(foreach sx,$(SUFFIXES),$(patsubst %$(sx), $($(1)_objdir)/%.o, $(filter %$(sx),$($(2)_SOURCES)))) +$(2)_SRCS = $(addprefix src/$(1)/jni/,$($(2)_SOURCES)) + +$($(1)_libdir)/$(LIB)$(2)$(SO): $$($(2)_OBJS) $($(2)_LIBADD) $($(2)_DEPENDENCIES) + @install -d $$(@D) + $($(TARGET)_CC) -o $$@ -shared \ + $($(TARGET)_LDFLAGS) $($(2)_LDFLAGS) $$($(2)_OBJS) $($(2)_LIBADD) $($(TARGET)_LDLIBS) $($(2)_LDLIBS) + +$($(1)_objdir)/%.o: src/$(1)/jni/%.c bin/status/$(1).classes + @install -d $$(@D) + $($(TARGET)_CC) -Isrc/$(1)/jni -Ibin/include/$(1) $($(TARGET)_CPPFLAGS) $($(2)_CPPFLAGS) \ + $($(TARGET)_CFLAGS) $($(2)_CFLAGS) -c -o $$@ $$< + +$($(1)_incdir)/%.h: src/$(1)/jni/%.h + install -D $$< $$@ + +$($(1)_objdir)/%.d: src/$(1)/jni/%.c bin/status/$(1).classes + @install -d $$(@D) + @rm -f $$@ + @$($(TARGET)_CC) -MM -MT "bin/$(1)/$(TARGET)/obj/$$*.o" -Isrc/$(1)/jni -Ibin/include/$(1) \ + $($(TARGET)_CPPFLAGS) $($(2)_CPPFLAGS) $$< -o $$@.d 2>/dev/null + @sed 's,\($$*\.o\) *:,\1 $$@ : ,g' $$@.d > $$@ ; rm $$@.d + +bin jni $(1) bin/$(1)/$(TARGET)/$(1).jmod: $($(1)_libdir)/$(LIB)$(2)$(SO) \ + $(addprefix $($(1)_incdir)/,$($(2)_HEADERS)) \ + $(addprefix $($(1)_bindir)/,$($(2)_COMMANDS)) + +$(if $(filter clean dist gen,$(MAKECMDGOALS)),,-include $$($(2)_OBJS:.o=.d)) +endef + +#$(foreach module,$(java_JMODS),$(foreach library,$($(module)_JNI_LIBRARIES),$(info $(call jni_library,$(module),$(library))))) +$(foreach module,$(java_JMODS),$(foreach library,$($(module)_JNI_LIBRARIES),$(eval $(call jni_library,$(module),$(library))))) + +# ###################################################################### + +dist: + @install -d bin + tar cfz bin/$(dist_NAME)-$(dist_VERSION).tar.gz \ + --transform=s,^,$(dist_NAME)-$(dist_VERSION)/, \ + config.make java.make Makefile src \ + $(dist_EXTRA) + diff --git a/junit.make b/junit.make new file mode 100644 index 0000000..7273f57 --- /dev/null +++ b/junit.make @@ -0,0 +1,90 @@ +# +# Copyright (C) 2019 Michael Zucchi +# +# This is the copyright for java.make +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# ###################################################################### +# junit(4) makefile fragment + +# This is still in development. + +# This should be compatible with the way the ant task (and netbeans) +# runs the same junit4 tests + +# It needs some parameterisation, e.g. module path, generated tests, +# etc. + +# Where to get junit4: +# http://central.maven.org/maven2/junit/junit/4.12/junit-4.12.jar +# http://central.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar + +# this is from netbeans 11 +junit4_classpath ?= /usr/local/netbeans-11.0/platform/modules/ext/junit-4.12.jar \ + /usr/local/netbeans-11.0/platform/modules/ext/hamcrest-core-1.3.jar +junit4_runner ?= org.junit.runner.JUnitCore + +# ###################################################################### + +define java_tests= +ifndef $(1)_TEST_JAVA +$(1)_TEST_JAVA := $$(shell find src/$(1)/tests -type f -name '*.java') +$(1)_TEST := $$(subst /,.,$$(basename $$(shell find src/$(1)/tests -type f -name '*Test.java' -printf '%P\n'))) +$(1)_IT := $$(subst /,.,$$(basename $$(shell find src/$(1)/tests -type f -name '*IT.java' -printf '%P\n'))) +endif + +test: $(1)-test +test-it: $(1)-it +check: $(1)-test $(1)-it +bin/status/$(1).tests: bin/status/$(1).classes $(notzed.dez_TEST_JAVA) +endef + +$(foreach module,$(java_MODULES),$(if $(wildcard src/$(module)/tests),$(eval $(call java_tests,$(module))))) + +# ###################################################################### + +bin/status/%.tests: + @install -d $(@D) + $(JAVAC) \ + -classpath $(subst $(S),:,$(junit4_classpath) ) \ + --patch-module $*=src/$*/tests \ + --add-reads $*=ALL-UNNAMED \ + -source 11 \ + --module-source-path 'src/*/tests' \ + --module-path bin/modules \ + -d bin/tests \ + $($*_TEST_JAVA) + touch $@ + +%-test: bin/status/%.tests + java \ + --class-path $(subst $(S),:,$(junit4_classpath)) \ + --module-path bin/modules \ + --patch-module $*=bin/tests/$* \ + --add-modules $* \ + --add-reads $*=ALL-UNNAMED \ + $(junit4_runner) \ + $($*_TEST) + +%-it: bin/status/%.tests + java \ + --class-path $(subst $(S),:,$(junit4_classpath)) \ + --module-path bin/modules \ + --patch-module $*=bin/tests/$* \ + --add-modules $* \ + --add-reads $*=ALL-UNNAMED \ + $(junit4_runner) \ + $($*_IT) diff --git a/nbproject/build-impl.xml b/nbproject/build-impl.xml new file mode 100644 index 0000000..4843ac9 --- /dev/null +++ b/nbproject/build-impl.xml @@ -0,0 +1,1737 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + self.setSelected(!new java.io.File(file, "module-info.java").exists()); + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @{paths} + + + + + + + + + + + + @{paths} + + + + + + + + + + + + + + + + + + + + Must set src.dir + Must set test.src.dir + Must set build.dir + Must set dist.dir + Must set build.modules.dir + Must set dist.javadoc.dir + Must set build.test.modules.dir + Must set build.test.results.dir + Must set build.classes.excludes + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Must set javac.includes + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + No tests executed. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Must set JVM to use for profiling in profiler.info.jvm + Must set profiler agent JVM arguments in profiler.info.jvmargs.agent + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + function coalesce(input, keyValueSeparator, multiSeparator, entrySeparator) { + var result = []; + var values = {}; + + (typeof input === "string" ? input.split(entrySeparator) : input).forEach(function(entry) { + var idx = entry.indexOf(keyValueSeparator); + if (idx < 1) { + result.push(entry); + } else { + var key = entry.substring(0, idx); + var val = entry.substring(idx + 1); + if (!values[key]) { + values[key] = []; + } + values[key].push(val.trim()); + } + }); + Object.keys(values).sort().forEach(function(k) { + result.push(k + keyValueSeparator + values[k].join(multiSeparator)); + }); + return result.join(" " + entrySeparator); + } + self.project.setProperty(attributes.get("property"), + coalesce(attributes.get("value"), + attributes.get("value-sep"), + attributes.get("entry-sep"), + attributes.get("multi-sep") + )); + + + + + + + + + function expandGroup(grp) { + var exp = []; + var item = ""; + var depth = 0; + + for (i = 0; i < grp.length; i++) { + var c = grp[i]; + switch (c) { + case '{': + if (depth++ === 0) { + continue; + } + break; + case '}': + if (--depth === 0) { + exp.push(item); + continue; + } + break; + case ',': + if (depth === 1) { + exp.push(item); + item = ""; + continue; + } + default: + break; + } + item = item + c; + } + return exp; + } + + function pathVariants(spec, res) { + res = res || []; + var start = spec.indexOf('{'); + if (start === -1) { + res.push(spec); + return res; + } + var depth = 1; + var end; + for (end = start + 1; end < spec.length && depth > 0; end++) { + var c = spec[end]; + switch (c) { + case '{': depth++; break; + case '}': depth--; break; + } + } + var prefix = spec.substring(0, start); + var suffix = spec.substring(end); + expandGroup(spec.slice(start, end)).forEach(function (item) { + pathVariants(prefix + item + suffix, res); + }) + return res; + } + + function toRegexp2(spec, filepattern, separator) { + var prefixes = []; + var suffixes = []; + pathVariants(spec).forEach(function(item) { + suffixes.push(item); + }); + var tail = ""; + var separatorString = separator; + if (separatorString == "\\") { + separatorString = "\\\\"; + } + if (filepattern && filepattern != tail) { + tail = separatorString + filepattern; + } + return "([^" + separatorString +"]+)\\Q" + separator + "\\E(" + suffixes.join("|") + ")" + tail; + } + self.project.setProperty(attributes.get("property"), + toRegexp2(attributes.get("modsource"), attributes.get("filepattern"), self.project.getProperty("file.separator"))); + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Must select some files in the IDE or set javac.includes + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + No main class specified + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Must select one file in the IDE or set run.class + + + + Must select one file in the IDE or set run.class + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Must select one file in the IDE or set debug.class + + + + + Must select one file in the IDE or set debug.class + + + + + Must set fix.includes + + + + + + + + + + + This target only works when run from inside the NetBeans IDE. + + + + + + + + + Must select one file in the IDE or set profile.class + This target only works when run from inside the NetBeans IDE. + + + + + + + + + This target only works when run from inside the NetBeans IDE. + + + + + + + + + + + + + This target only works when run from inside the NetBeans IDE. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Must select one file in the IDE or set run.class + + + + + + Must select some files in the IDE or set test.includes + + + + + Must select one file in the IDE or set run.class + + + + + Must select one file in the IDE or set applet.url + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + self.setSelected(!new java.io.File(file, "module-info.class").exists()); + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Must select some files in the IDE or set javac.includes + + + + + + + + + + + + + + + + + + + + + + + + + + Some tests failed; see details above. + + + + + + + + + Must select some files in the IDE or set test.includes + + + + Some tests failed; see details above. + + + + Must select some files in the IDE or set test.class + Must select some method in the IDE or set test.method + + + + Some tests failed; see details above. + + + + + Must select one file in the IDE or set test.class + + + + Must select one file in the IDE or set test.class + Must select some method in the IDE or set test.method + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/nbproject/genfiles.properties b/nbproject/genfiles.properties new file mode 100644 index 0000000..651e80a --- /dev/null +++ b/nbproject/genfiles.properties @@ -0,0 +1,8 @@ +build.xml.data.CRC32=508a6882 +build.xml.script.CRC32=b55362bc +build.xml.stylesheet.CRC32=32069288@1.6.1 +# This file is used by a NetBeans-based IDE to track changes in generated files such as build-impl.xml. +# Do not edit this file. You may delete it but then the IDE will never regenerate such files for you. +nbproject/build-impl.xml.data.CRC32=e4239d52 +nbproject/build-impl.xml.script.CRC32=4564887e +nbproject/build-impl.xml.stylesheet.CRC32=0f0529df@1.6.1 diff --git a/nbproject/project.properties b/nbproject/project.properties new file mode 100644 index 0000000..1702468 --- /dev/null +++ b/nbproject/project.properties @@ -0,0 +1,97 @@ +annotation.processing.enabled=true +annotation.processing.enabled.in.editor=false +annotation.processing.processors.list= +annotation.processing.run.all.processors=true +annotation.processing.source.output=${build.generated.sources.dir}/ap-source-output +application.title=notzed.dez +application.vendor=notzed +build.classes.dir=${build.dir}/classes +build.classes.excludes=**/*.java,**/*.form +# This directory is removed when the project is cleaned: +build.dir=build +build.generated.dir=${build.dir}/generated +build.generated.sources.dir=${build.dir}/generated-sources +build.modules.dir=${build.dir}/modules +# Only compile against the classpath explicitly listed here: +build.sysclasspath=ignore +build.test.classes.dir=${build.dir}/test/classes +build.test.modules.dir=${build.dir}/test/modules +build.test.results.dir=${build.dir}/test/results +# Uncomment to specify the preferred debugger connection transport: +#debug.transport=dt_socket +debug.classpath=\ + ${run.classpath} +debug.modulepath=\ + ${run.modulepath} +debug.test.classpath=\ + ${run.test.classpath} +debug.test.modulepath=\ + ${run.test.modulepath} +# Files in build.classes.dir which should be excluded from distribution jar +dist.archive.excludes= +# This directory is removed when the project is cleaned: +dist.dir=dist +dist.javadoc.dir=${dist.dir}/javadoc +dist.jlink.dir=${dist.dir}/jlink +dist.jlink.output=${dist.jlink.dir}/notzed.dez +endorsed.classpath= +excludes= +includes=** +jar.compress=false +javac.classpath= +# Space-separated list of extra javac options +javac.compilerargs=-Xlint:unchecked +javac.deprecation=false +javac.external.vm=false +javac.modulepath= +javac.processormodulepath= +javac.processorpath=\ + ${javac.classpath} +javac.source=11 +javac.target=11 +javac.test.classpath=\ + ${javac.classpath}:\ + ${libs.junit_4.classpath}:\ + ${libs.hamcrest.classpath} +javac.test.modulepath=\ + ${javac.modulepath}:\ + ${build.modules.dir} +javac.test.processorpath=\ + ${javac.test.classpath} +javadoc.additionalparam= +javadoc.author=false +javadoc.encoding=${source.encoding} +javadoc.html5=false +javadoc.noindex=false +javadoc.nonavbar=false +javadoc.notree=false +javadoc.private=false +javadoc.splitindex=true +javadoc.use=true +javadoc.version=false +javadoc.windowtitle= +# The jlink additional root modules to resolve +jlink.additionalmodules= +# The jlink additional command line parameters +jlink.additionalparam= +jlink.launcher=true +jlink.launcher.name=notzed.dez +platform.active=default_platform +run.classpath= +# Space-separated list of JVM arguments used when running the project. +# You may also define separate properties like run-sys-prop.name=value instead of -Dname=value. +# To set system properties for unit tests define test-sys-prop.name=value: +run.jvmargs= +run.modulepath=\ + ${javac.modulepath}:\ + ${build.modules.dir} +run.test.classpath=\ + ${javac.test.classpath} +run.test.modulepath=\ + ${javac.test.modulepath}:\ + ${build.test.modules.dir} +source.encoding=UTF-8 +src.dir=src +src.dir.path=classes +test.src.dir=src +test.src.dir.path=tests diff --git a/nbproject/project.xml b/nbproject/project.xml new file mode 100644 index 0000000..6722575 --- /dev/null +++ b/nbproject/project.xml @@ -0,0 +1,25 @@ + + + org.netbeans.modules.java.j2semodule + + + notzed.dez + + + + + + + + + + notzed_nativez + jar + + jar + clean + notzed.nativez.jar + + + + diff --git a/src/notzed.dez.demo/classes/au/notzed/dez/demo/DEZPrinter.java b/src/notzed.dez.demo/classes/au/notzed/dez/demo/DEZPrinter.java new file mode 100644 index 0000000..e64833e --- /dev/null +++ b/src/notzed.dez.demo/classes/au/notzed/dez/demo/DEZPrinter.java @@ -0,0 +1,236 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez.demo; + +import au.notzed.dez.DEZFormat; +import java.io.PrintStream; + +/** + * Prints a DEZ patch. + *

+ * Displays the editing commands and some stats. + *

+ * @see au.notzed.dez.DEZEncoder + */ +public class DEZPrinter implements DEZFormat { + + private final byte[] patch; + private int pi, si; + // Recent address cache. TODO: These can be combined + private final int[] matchAddr = new int[64]; + private int matchNext = 0; + private int[] recentAddr = new int[32]; + private int recentNext = 0; + // some statistics. + private Stats copy = new Stats(); + private Stats add = new Stats(); + private Stats run = new Stats(); + + public DEZPrinter(byte[] patch) { + this.patch = patch; + } + + private int decodeInt() { + int v = 0; + byte b; + + do { + b = patch[pi++]; + v = (v << 7) | (b & 0x7f); + } while ((b & 0x80) != 0); + + return v; + } + + private void updateAddr(int addr) { + recentAddr[recentNext++] = addr; + recentNext &= recentAddr.length - 1; + matchAddr[matchNext++] = addr; + matchNext &= matchAddr.length - 1; + } + + private int decodeAddr() { + int addr; + + int op = patch[pi]; + + if ((op & 0x80) == 0) { + // An encoded address + if ((op & 0x40) == 0) { + //direct + pi++; + addr = matchAddr[op & 63]; + } else { + // signed relative + pi++; + int d = decodeInt(); + if ((op & 0x20) != 0) + d = -d; + addr = recentAddr[op & 31] + d; + } + } else { + // Normal address + addr = decodeInt(); + } + updateAddr(addr); + return addr; + } + + /** + * Dumps the patch details to stdout. + * + */ + public void print() { + DEZPrinter.this.print(System.out); + } + + public void print(PrintStream out) { + int ti = 0; + int flags; + int smallest = 4; + + pi = 0; + si = 0; + + // 'decode' magic + out.printf("magic: %c%c%c%c\n", patch[0], patch[1], patch[2], patch[3]); + pi += 4; + // decode flags + flags = patch[pi++]; + out.printf("flags: %02x\n", flags & 0xff); + + if ((flags & DEZ_SMALLEST) != 0) + smallest = decodeInt(); + out.printf("smallest: %d\n", smallest); + + // get sizes + int sourceSize = decodeInt(); + int targetSize = decodeInt(); + + int oc = 0; + + out.printf("source: %d\ntarget: %d\n", sourceSize, targetSize, patch.length); + + while (ti < targetSize) { + int op = patch[pi++] & 0xff; + + out.printf("%02x ", op); + + if ((op & 0x80) == 0) { + // One-byte pair commands + if ((op & 0x40) == 0) { + // 00AAACCC + int add = ((op >> 3) & 0x07) + 1; + int copy = (op & 0x07) + smallest; + int addr; + + pi += add; + addr = decodeAddr(); + + out.printf("%08x: ADD %d COPY %d @ $%08x\n", + ti, add, copy, addr); + ti += add + copy; + + this.add.add(add); + this.copy.add(copy); + } else { + // 01cccCCC + int copy0 = ((op >> 3) & 0x07) + smallest; + int copy1 = (op & 0x07) + smallest; + int addr0 = decodeAddr(); + int addr1 = decodeAddr(); + + out.printf("%08x: COPY %d @ $%08x COPY %d @ $%08x\n", + ti, copy0, addr0, copy1, addr1); + ti += copy0 + copy1; + + this.copy.add(copy0); + this.copy.add(copy1); + } + } else { + // 1NNNNNNN + // 0 ... 99 copy smallest ... 99+smallest + // 100 ... 123 add 1 ... 24 + // 124 copy + length + // 125 add + length + // 126 run + length + byte + // 127 ext command + int n = op & 0x7f; + + if (n < OP_SINGLE_SPLIT) { + // COPY + addr + int copy = n + smallest; + int addr = decodeAddr(); + + out.printf("%08x: COPY %d @ $%08x\n", ti, copy, addr); + ti += copy; + + this.copy.add(copy); + } else if (n < 124) { + // ADD + data + int add = n - OP_SINGLE_SPLIT + 1; + + out.printf("%08x: ADD %d\n", ti, add); + ti += add; + pi += add; + + this.add.add(add); + } else if (n == 124) { + // COPY + length + addr + int copy = decodeInt() + smallest + OP_SINGLE_SPLIT; + int addr = decodeAddr(); + + out.printf("%08x: COPY %d @ $%08x\n", ti, copy, addr); + ti += copy; + + this.copy.add(copy); + } else if (n == 125) { + // ADD + length + data + int add = decodeInt() + (124 - OP_SINGLE_SPLIT) + 1; + + out.printf("%08x: ADD %d\n", ti, add); + ti += add; + pi += add; + + this.add.add(add); + } else if (n == 126) { + // RUN + length + byte + int run = decodeInt() + 3; + byte r = patch[pi++]; + + out.printf("%08x: RUN $%02x %d\n", ti, r & 0xff, run); + ti += run; + + this.run.add(run); + } + } + } + if (ti > targetSize) + throw new ArrayIndexOutOfBoundsException(String.format("Target overflow %d > %d", ti, targetSize)); + if (ti != targetSize) + throw new ArrayIndexOutOfBoundsException(String.format("Target short write %d != %d", ti, targetSize)); + + out.printf("summary\n"); + out.printf(" copy: %s\n", copy); + out.printf(" add: %s\n", add); + out.printf(" run: %s\n", run); + out.printf("patch: %d\n", pi); + out.printf("sorce: %d\n", sourceSize); + out.printf("targt: %d\n", targetSize); + } + +} diff --git a/src/notzed.dez.demo/classes/au/notzed/dez/demo/PrintEncoder.java b/src/notzed.dez.demo/classes/au/notzed/dez/demo/PrintEncoder.java new file mode 100644 index 0000000..7f398bf --- /dev/null +++ b/src/notzed.dez.demo/classes/au/notzed/dez/demo/PrintEncoder.java @@ -0,0 +1,52 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez.demo; + +import au.notzed.dez.ByteDeltaEncoder; + +/** + * A dummy encoder for debugging purposes. + */ +public class PrintEncoder implements ByteDeltaEncoder { + + int oc = 0; + + @Override + public void init(int sourceSize, int targetSize) { + System.out.printf("source = %d, target=%d\n", sourceSize, targetSize); + } + + @Override + public void copy(int addr, int len) { + System.out.printf("%5d: COPY @ %d for %d\n", oc++, addr, len); + } + + @Override + public void add(byte[] data, int off, int len) { + System.out.printf("%5d: ADD @ %d for %d\n", oc++, off, len); + } + + @Override + public void run(byte b, int len) { + System.out.printf("%5d: RUN $%02x for %d\n", oc++, b & 0xff, len); + } + + @Override + public byte[] toPatch() { + return new byte[0]; + } +} diff --git a/src/notzed.dez.demo/classes/au/notzed/dez/demo/Stats.java b/src/notzed.dez.demo/classes/au/notzed/dez/demo/Stats.java new file mode 100644 index 0000000..f7b78d8 --- /dev/null +++ b/src/notzed.dez.demo/classes/au/notzed/dez/demo/Stats.java @@ -0,0 +1,66 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez.demo; + +/** + * Basic incremental stats accumulator. + *

+ * This may be used as a streaming collector/reducer. + */ +public class Stats { + + int min = Integer.MAX_VALUE, max = Integer.MIN_VALUE; + long sum = 0; + int count = 0; + + public void add(int v) { + sum += v; + count += 1; + min = Math.min(v, min); + max = Math.max(v, max); + } + + public void addAll(Stats o) { + sum += o.sum; + count += o.count; + min = Math.min(o.min, min); + max = Math.min(o.max, max); + } + + public long getSum() { + return sum; + } + + public int getMax() { + return max; + } + + public int getMin() { + return min; + } + + public double getMean() { + return (double) sum / count; + } + + @Override + public String toString() { + if (count == 0) + return "min/max/ave = 0 / 0 / 0"; + return String.format("min/max/ave = %d / %d / %f", min, max, (double) sum / count); + } +} diff --git a/src/notzed.dez.demo/classes/au/notzed/dez/demo/dez.java b/src/notzed.dez.demo/classes/au/notzed/dez/demo/dez.java new file mode 100644 index 0000000..28302ad --- /dev/null +++ b/src/notzed.dez.demo/classes/au/notzed/dez/demo/dez.java @@ -0,0 +1,118 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez.demo; + +import au.notzed.dez.ByteDeltaEncoder; +import au.notzed.dez.ByteMatcherHash; +import au.notzed.dez.DEZDecoder; +import au.notzed.dez.DEZEncoder; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Paths; + +/** + * A basic tool. + *

+ * Creating a delta: + * java -cp au.notzed.dez.DEZ -c source target > patch + *

+ * Restoring a delta: + * java -cp au.notzed.dez.DEZ -d source patch > restored + *

+ * Printing delta: + * java -cp au.notzed.dez.DEZ -p patch + * + */ +public class dez { + + static void usage() { + System.err.println("Usage:\n"); + System.err.println(" create:\n dez -c source target > patch"); + System.err.println(" decode:\n dez -d source patch > target"); + System.err.println(" print:\n dez -p patch"); + System.exit(1); + } + + static void create(String source, String target) { + try { + byte[] s = Files.readAllBytes(Paths.get(source)); + byte[] t = Files.readAllBytes(Paths.get(target)); + + byte[] p = ByteDeltaEncoder.toDiff(new ByteMatcherHash(6, 4, s, 1, t), new DEZEncoder()); + + System.out.write(p); + } catch (IOException ex) { + System.err.printf("IOError: %s\n", ex.getLocalizedMessage()); + } + } + + static void decode(String source, String patch) { + try { + byte[] s = Files.readAllBytes(Paths.get(source)); + byte[] p = Files.readAllBytes(Paths.get(patch)); + + byte[] t = new DEZDecoder(s, p).decode(); + + System.out.write(t); + } catch (IOException ex) { + System.err.printf("IOError: %s\n", ex.getLocalizedMessage()); + } + } + + static void print(String patch) { + try { + byte[] p = Files.readAllBytes(Paths.get(patch)); + + new DEZPrinter(p).print(); + } catch (IOException ex) { + System.err.printf("IOError: %s\n", ex.getLocalizedMessage()); + } + } + + public static void main(String[] args) { + if (args.length < 2) { + usage(); + return; + } + + switch (args[0]) { + case "-c": + if (args.length == 3) + create(args[1], args[2]); + else + usage(); + break; + case "-d": + if (args.length == 3) + decode(args[1], args[2]); + else + usage(); + break; + case "-p": + if (args.length == 2) + print(args[1]); + else + usage(); + break; + default: + usage(); + break; + } + + } + +} diff --git a/src/notzed.dez.demo/classes/module-info.java b/src/notzed.dez.demo/classes/module-info.java new file mode 100644 index 0000000..41da0cf --- /dev/null +++ b/src/notzed.dez.demo/classes/module-info.java @@ -0,0 +1,22 @@ +/* + * Copyright (C) 2019 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +module notzed.dez.demo { + requires transitive notzed.dez; + + exports au.notzed.dez.demo; +} diff --git a/src/notzed.dez/classes/au/notzed/dez/ByteDeltaEncoder.java b/src/notzed.dez/classes/au/notzed/dez/ByteDeltaEncoder.java new file mode 100644 index 0000000..a3546a1 --- /dev/null +++ b/src/notzed.dez/classes/au/notzed/dez/ByteDeltaEncoder.java @@ -0,0 +1,101 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + +/** + * The interface for encoding a delta. + *

+ * A delta encoder will implement a specific file/transfer format. + */ +public interface ByteDeltaEncoder { + + /** + * Initialises creating a new patch. + * + * @param sourceSize + * @param targetSize + */ + public void init(int sourceSize, int targetSize); + + /** + * Appends a copy command. + * + * @param addr + * @param len + */ + public void copy(int addr, int len); + + /** + * Appends an append command. + * + * @param data + * @param off + * @param len + */ + public void add(byte[] data, int off, int len); + + /** + * Appends a byte-run. + * + * @param b + * @param len + */ + public void run(byte b, int len); + + /** + * Retrieves the patch. + * + * @return + */ + public byte[] toPatch(); + + /** + * Creates a delta from a matcher and writes it to an encoder. + * + * @param matcher + * @param enc + * @return + */ + public static byte[] toDiff(ByteMatcher matcher, ByteDeltaEncoder enc) { + byte[] source = matcher.getSource(); + byte[] target = matcher.getTarget(); + + enc.init(source.length, target.length); + + int targetEnd = 0; + int state; + + while ((state = matcher.nextMatch()) != ByteMatcher.EOF) { + int toff = matcher.getTargetOffset(); + int slength = matcher.getLength(); + + if (targetEnd != toff) + enc.add(target, targetEnd, toff - targetEnd); + + if (state == ByteMatcher.RUN) + enc.run(matcher.getRunByte(), slength); + else + enc.copy(matcher.getMatchOffset(), slength); + + targetEnd = toff + slength; + } + if (targetEnd != target.length) + enc.add(target, targetEnd, target.length - targetEnd); + + return enc.toPatch(); + } +} diff --git a/src/notzed.dez/classes/au/notzed/dez/ByteMatcher.java b/src/notzed.dez/classes/au/notzed/dez/ByteMatcher.java new file mode 100644 index 0000000..fcc1c2a --- /dev/null +++ b/src/notzed.dez/classes/au/notzed/dez/ByteMatcher.java @@ -0,0 +1,109 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + +/** + * Common interface for byte matchers. + *

+ * Byte matchers look for common sub-strings between a source and + * a target byte array and may optionally detect runs of duplicated + * bytes. + */ +public interface ByteMatcher { + + public final static int COPY = 0; + public final static int RUN = 1; + public static final int EOF = -1; + + /** + * Finds the next match or run. + *

+ * Note that only matches or byte runs will be indicated. The location + * of non-matching data (i.e. append sequences) must be determined from + * the difference between the last targetOffset, the last length, and the + * current targetOffset. + *

+ * + * @return the new state. + */ + public int nextMatch(); + + /** + * Retrieves the current target position. + *

+ * The position within the target to which the current match refers. + * + * @return + */ + public int getTargetOffset(); + + /** + * Retrieves the best match location. + *

+ * If the current state is COPY then this returns a valid location + * of the best match. This should be interpreted + * using {@link #getBlockArray} and {@link #getBlockOffset}. + * + * @return + */ + public int getMatchOffset(); + + /** + * Retrieves the byte to be run-length encoded. + *

+ * If the current state is RUN then this returns the corresponding byte to run. + * + * @return + */ + public byte getRunByte(); + + /** + * Retrieves the current length. + *

+ * This is the number of bytes to copy for the COPY state or repeat for the RUN state. + * + * @return + */ + public int getLength(); + + /** + * Retrieves the array containing the current match. + *

+ * Maps the offset to the correct internal array. + * + * @param offset + * @return + * @see getBlockOffset + */ + public byte[] getBlockArray(int offset); + + /** + * Calculates the offset for the block array. + *

+ * Maps the match offset to the array from getBlockArray. + * + * @param offset + * @return + * @see getBlockArray + */ + public int getBlockOffset(int offset); + + public byte[] getSource(); + + public byte[] getTarget(); + +} diff --git a/src/notzed.dez/classes/au/notzed/dez/ByteMatcherHash.java b/src/notzed.dez/classes/au/notzed/dez/ByteMatcherHash.java new file mode 100644 index 0000000..1cfd2fd --- /dev/null +++ b/src/notzed.dez/classes/au/notzed/dez/ByteMatcherHash.java @@ -0,0 +1,342 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + +import java.util.Arrays; + +/** + * Finds common strings of bytes between a source and target buffer. + *

+ * This is basically an implementation of Bentley & McIllroy's paper + * ``Data Compression Using Long Common Strings'' applied instead to producing + * deltas and using a cyclic hash as the fingerprint function. + *

+ * Two other + * modifications are that back-tracking is not implemented but instead + * overlapping blocks can be used by setting the step size. + * And a further refinement is the detection of runs of the same byte which + * might otherwise pollute the hash tree for certain data. + */ +public class ByteMatcherHash implements ByteMatcher { + + private final int b; + private final int shortest; + private final byte[] source; + private final int sstep; + private final byte[] target; + // Incremental hashes + private final CyclicHash targetHash; + private final CyclicHash sourceHash; + // Runtime state + private int ti; + private int thash; + private int skipTo; + private int targetAvailable; + // Public state + private int bestLength; + private int bestOffset; + private int targetOffset; + private byte runByte; + // Data to implement chain limit. + private int limit = Integer.MAX_VALUE; + private int roll = 1; + + /** + * Inline hash+array table. + *

+ * All values which hash the same are appended to the same list. + *

+ * Index is the current length/next insertion point for that hash chain. + * Values contains the chained hash table values. + */ + final private int hashMask; + final private int[][] hashValues; + + /** + * Creates and initialises a new byte matcher. + *

+ * This is a single-use object. + *

+ * A step size of 1 produces the best output but requires the most memory and run time. + *

+ * @param b Sets block size, which is the number of bytes hashed per key (>=3). + * @param shortest shortest string considered for a copy. Typically 4 bytes but dependent on the encoder used and + * the value of b. + * @param source Source array. + * @param sstep Sets the step size which is the interval of sampling of the source. + * @param target Target array. + */ + public ByteMatcherHash(int b, int shortest, byte[] source, int sstep, byte[] target) { + int size; + + b = Math.max(b, 3); + + // This may need tuning. + int logN = 31 - Integer.numberOfLeadingZeros((source.length + target.length) / sstep); + size = 1 << Math.max(14, logN - 5); + + hashMask = size - 1; + hashValues = new int[size][]; + + targetHash = new CyclicHash(b); + sourceHash = new CyclicHash(b); + + this.b = b; + this.shortest = shortest; + this.source = source; + this.sstep = sstep; + this.target = target; + + addAll(source, source.length, 0, 0); + if (target.length >= b) + this.thash = targetHash.init(target, 0); + } + + /** + * Checks for run of 3 bytes. + *

+ * Boundaries are not checked. + * + * @param s + * @param pos + * @return + */ + private boolean isRun(byte[] s, int pos) { + byte v = s[pos]; + return v == s[pos + 1] && v == s[pos + 2]; + } + + private int addAll(byte[] s, int limit, int pos, int off) { + if (sstep == 1) { + if (pos == 0 && limit >= b) { + add(sourceHash.init(s, 0), off); + pos = 1; + } + + while (pos <= limit - b) { + int hash = sourceHash.update(s[pos - 1], s[pos - 1 + b]); + + if (!isRun(s, pos)) + add(hash, pos + off); + pos += 1; + } + } else { + while (pos <= limit - b) { + if (!isRun(s, pos)) + add(sourceHash.init(s, pos), pos + off); + pos += sstep; + } + } + return pos; + } + + /** + * Sets the hash bucket chain limit. + *

+ * Limiting this value will reduce the search for common prefixes + * which will in turn limit the search accuracy. A runtime/quality + * trade-off. + *

+ * By default there is no limit. + * + * @param value A value between 2 and 30 which sets the length limit to (1<<value)-1. + */ + public void setChainLimit(int value) { + if (value >= 31) + this.limit = Integer.MAX_VALUE; + else + this.limit = (1 << Math.max(2, value)); + } + + private void add(int hash, int value) { + int j = hash & hashMask; + int[] vs = hashValues[j]; + + if (vs == null) { + hashValues[j] = vs = new int[4]; + vs[0] = 2; + vs[1] = value; + } else { + int i = vs[0]; + + if (i < limit) { + if (i >= vs.length) + hashValues[j] = vs = Arrays.copyOf(vs, vs.length * 2); + vs[i++] = value; + vs[0] = i; + } else { + int ri = (roll++) & (vs.length - 1); + + if (ri == 0) + ri = 1; + vs[ri] = value; + + //vs[roll++] = value; + //if (roll == vs.length) + // roll = 1; + } + } + } + + /** + * Finds the length of similarity between the two sub-arrays. + *

+ * + * @param soff source offset starting location, locations above source.length refer to the target buffer. + * @param toff target offset starting location + * @param bestLength sets the current best length. Used to short-circuit 'definitely can't be longer' cases. + * @return how many bytes are sequentially identical. + */ + private int matchLength(int soff, int toff, int bestLength) { + if (soff < source.length) { + int limit = Math.min(source.length - soff, target.length - toff); + + if (limit < bestLength + || (limit > bestLength && source[soff + bestLength] != target[toff + bestLength])) + return 0; + + for (int i = 0; i < limit; i++) + if (source[soff + i] != target[toff + i]) + return i; + return limit; + } else { + soff -= source.length; + int limit = Math.min(target.length - soff, target.length - toff); + + if (limit < bestLength + || (limit > bestLength && target[soff + bestLength] != target[toff + bestLength])) + return 0; + + for (int i = 0; i < limit; i++) + if (target[soff + i] != target[toff + i]) + return i; + return limit; + } + } + + @Override + public byte[] getSource() { + return source; + } + + @Override + public byte[] getTarget() { + return target; + } + + @Override + public int getMatchOffset() { + return bestOffset; + } + + @Override + public int getTargetOffset() { + return targetOffset; + } + + @Override + public int getLength() { + return bestLength; + } + + @Override + public byte getRunByte() { + return runByte; + } + + @Override + public int nextMatch() { + bestLength = 0; + bestOffset = 0; + + /** + * Reset thash on seek. + */ + if (skipTo != ti) { + if (skipTo <= target.length - b) + thash = targetHash.init(target, skipTo); + ti = skipTo; + } + + while (bestLength < shortest && ti <= target.length - b) { + /** + * Short circuit test for byte-runs. + */ + if (isRun(target, ti)) { + byte b0 = target[ti]; + int j = ti + 3; + while (j < target.length && target[j] == b0) + j++; + targetOffset = ti; + bestLength = j - ti; + runByte = b0; + skipTo = j; + return RUN; + } + + /** + * Include any of the target buffer which has been decoded to this point to the hash table. + */ + targetAvailable = addAll(target, ti + b - 1, targetAvailable, source.length); + + /** + * Checks the current string for the longest match against the hash table. + */ + int j = thash & hashMask; + int[] soffs = hashValues[j]; + + if (soffs != null) { + int len = soffs[0]; + + for (int i = 1; i < len; i++) { + int soff = soffs[i]; + int length = matchLength(soff, ti, bestLength); + + if (length > bestLength) { + bestLength = length; + bestOffset = soff; + } + } + } + + /** + * Advance. thash is always the next block to examine. + */ + targetOffset = ti; + ti += 1; + if (ti <= target.length - b) + thash = targetHash.update(target[ti - 1], target[ti - 1 + b]); + } + + if (bestLength >= shortest) { + skipTo = targetOffset + bestLength; + return COPY; + } else + return EOF; + } + + @Override + public byte[] getBlockArray(int offset) { + return (offset < source.length) ? source : target; + } + + @Override + public int getBlockOffset(int offset) { + return (offset < source.length) ? offset : offset - source.length; + } + +} diff --git a/src/notzed.dez/classes/au/notzed/dez/CyclicHash.java b/src/notzed.dez/classes/au/notzed/dez/CyclicHash.java new file mode 100644 index 0000000..70017f2 --- /dev/null +++ b/src/notzed.dez/classes/au/notzed/dez/CyclicHash.java @@ -0,0 +1,87 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + + +import static java.lang.Integer.rotateLeft; +import java.util.Random; + +/** + * Cyclic polynomial rolling hash. + *

+ * This implements a rolling hash of a fixed length. + *

+ * Input bytes are hashed using a random table. The randomness + * affects the quality of the hash. + */ +public class CyclicHash { + + private static final int[] random; + + private final int b; + private int hash; + private final int first; + private final int bits = 9; + + static { + // keyboard bashed the results unvalidated. + random = new Random(97435).ints(256).toArray(); + } + + /** + * Creates a cyclic hash. + * + * @param b + */ + public CyclicHash(int b) { + this.b = b; + this.first = ((b - 1) * bits) & 31; + } + + /** + * Initialises the hash. + *

+ * This will hash a block of data at the given location. + * + * @param data + * @param off + * @return + */ + public int init(byte[] data, int off) { + hash = 0; + for (int i = 0; i < b; i++) + hash = rotateLeft(hash, bits) ^ random[data[i + off] & 0xff]; + return hash; + } + + /** + * Updates the hash incrementally. + *

+ * Advance the hash by one location. + * + * @param leave the byte leaving. Must match the oldest byte included in the hash value. + * @param enter the byte entering. + * @return + */ + public int update(byte leave, byte enter) { + int leaving = rotateLeft(random[leave & 0xff], first); + int entering = random[enter & 0xff]; + + hash = rotateLeft(hash ^ leaving, bits) ^ entering; + return hash; + } +} diff --git a/src/notzed.dez/classes/au/notzed/dez/DEZDecoder.java b/src/notzed.dez/classes/au/notzed/dez/DEZDecoder.java new file mode 100644 index 0000000..19549db --- /dev/null +++ b/src/notzed.dez/classes/au/notzed/dez/DEZDecoder.java @@ -0,0 +1,227 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + +import java.io.PrintStream; +import java.util.Arrays; + +/** + * Decodes a DEZ patch. + *

+ * @see au.notzed.dez.DEZEncoder + */ +public class DEZDecoder implements DEZFormat { + + final static boolean dump = false; + + private final byte[] source; + private final byte[] patch; + private int pi, si; + // recent address cache. TODO: These can be combined + private int[] matchAddr = new int[64]; + private int matchNext = 0; + private int[] recentAddr = new int[32]; + private int recentNext = 0; + + public DEZDecoder(byte[] source, byte[] patch) { + this.source = source; + this.patch = patch; + } + + private int decodeInt() { + int v = 0; + byte b; + + do { + b = patch[pi++]; + v = (v << 7) | (b & 0x7f); + } while ((b & 0x80) != 0); + + return v; + } + + private int copy(int addr, byte[] target, int ti, int copy) { + if (addr < source.length) + System.arraycopy(source, addr, target, ti, copy); + else { + addr = addr - source.length; + for (int i = 0; i < copy; i++) + target[ti + i] = target[addr + i]; + } + return ti + copy; + } + + private void updateAddr(int addr) { + recentAddr[recentNext++] = addr; + recentNext &= recentAddr.length - 1; + matchAddr[matchNext++] = addr; + matchNext &= matchAddr.length - 1; + } + + private int decodeAddr() { + int addr; + int op = patch[pi]; + + if ((op & 0x80) == 0) { + // An encoded address + if ((op & 0x40) == 0) { + //direct + pi++; + addr = matchAddr[op & 63]; + } else { + // signed relative + pi++; + int d = decodeInt(); + if ((op & 0x20) != 0) + d = -d; + addr = recentAddr[op & 31] + d; + //updateAddr(addr); + } + } else { + // Normal address + addr = decodeInt(); + //updateAddr(addr); + } + updateAddr(addr); + + return addr; + } + + /** + * Recreates the original target data from the source and patch. + * + * @return + */ + public byte[] decode() { + int ti = 0; + int smallest = 4; + int flags; + + PrintStream out = dump ? System.out : null; + pi = 0; + si = 0; + + // 'decode' magic + //out.printf("magic: %c%c%c%c\n", patch[0] & 0xff, patch[1] & 0xff, patch[2] & 0xff, patch[3] & 0xff); + pi += 4; + // decode flags + flags = patch[pi++]; + if ((flags & DEZ_SMALLEST) != 0) + smallest = decodeInt(); + + // get sizes + int sourceSize = decodeInt(); + int targetSize = decodeInt(); + + int oc = 0; + + byte[] target = new byte[targetSize]; + + while (ti < targetSize) { + int op = patch[pi++] & 0xff; + + //out.printf("%02x ", op); + if ((op & 0x80) == 0) { + // One-byte pair commands + if ((op & 0x40) == 0) { + // 00AAACCC + int add = (op >> 3) + 1; + int copy = (op & 0x07) + smallest; + int addr; + + System.arraycopy(patch, pi, target, ti, add); + ti += add; + pi += add; + + addr = decodeAddr(); + if (dump) + out.printf("%08x: ADD %d COPY %d @ $%08x\n", ti, add, copy, addr); + ti = copy(addr, target, ti, copy); + } else { + // 01cccCCC + int copy0 = ((op >> 3) & 0x07) + smallest; + int copy1 = (op & 0x07) + smallest; + int addr0 = decodeAddr(); + int addr1 = decodeAddr(); + + ti = copy(addr0, target, ti, copy0); + ti = copy(addr1, target, ti, copy1); + + if (dump) + out.printf("%08x: COPY %d @ $%08x COPY %d @ $%08x\n", ti, copy0, addr0, copy1, addr1); + } + } else { + // 1NNNNNNN + // 0 ... 99 copy smallest ... 99+smallest + // 100 ... 123 add 1 ... 24 + // 124 copy + length + // 125 add + length + // 126 run + length + byte + // 127 ext command + int n = op & 0x7f; + + if (n < OP_SINGLE_SPLIT) { + // COPY + addr + int copy = n + smallest; + int addr = decodeAddr(); + + if (dump) + out.printf("%08x: COPY %d @ $%08x\n", ti, copy, addr); + ti = copy(addr, target, ti, copy); + } else if (n < OP_SINGLE_LIMIT) { + // ADD + data + int add = n - OP_SINGLE_SPLIT + 1; + + if (dump) + out.printf("%08x: ADD %d\n", ti, add); + System.arraycopy(patch, pi, target, ti, add); + ti += add; + pi += add; + } else if (n == OP_COPY) { + // COPY + length + addr + int copy = decodeInt() + smallest + OP_SINGLE_SPLIT; + int addr = decodeAddr(); + + if (dump) + out.printf("%08x: OP_COPY %d @ $%08x\n", ti, copy, addr); + ti = copy(addr, target, ti, copy); + } else if (n == OP_ADD) { + // ADD + length + data + int add = decodeInt() + (124 - OP_SINGLE_SPLIT) + 1; + + if (dump) + out.printf("%08x: OP_ADD %d\n", ti, add); + System.arraycopy(patch, pi, target, ti, add); + ti += add; + pi += add; + } else if (n == OP_RUN) { + // RUN + length + byte + int run = decodeInt() + 3; + byte r = patch[pi++]; + + if (dump) + out.printf("%08x: OP_RUN $%02x %d\n", ti, r & 0xff, run); + Arrays.fill(target, ti, ti + run, r); + ti += run; + } + } + } + + return target; + } + +} diff --git a/src/notzed.dez/classes/au/notzed/dez/DEZEncoder.java b/src/notzed.dez/classes/au/notzed/dez/DEZEncoder.java new file mode 100644 index 0000000..a298843 --- /dev/null +++ b/src/notzed.dez/classes/au/notzed/dez/DEZEncoder.java @@ -0,0 +1,318 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.util.Arrays; + +/** + * 'DeltaZ-1' format encoder. + *

+ * Encoder for the a binary delta format. + *

+ * + * @see au.notzed.dez.DEZDecoder + */ +public class DEZEncoder implements ByteDeltaEncoder, DEZFormat { + + final static boolean dump = false; + private final ByteArrayOutputStream patch = new ByteArrayOutputStream(); + + int here; + int header; + int sourceSize, targetSize; + + // Configurable parameters. + final int smallest; + // Last operation information + int last = -1; + byte[] lastData; + int lastOff; + int lastLen; + int lastAddr; + // Recent address cache. These can be combined? + private int[] matchAddr = new int[64]; + private int matchNext = 0; + private int[] recentAddr = new int[32]; + private int recentNext = 0; + + /** + * Creates a new encoder. + *

+ * The smallest copy size allowed is 4. + */ + public DEZEncoder() { + this.smallest = 4; + } + + /** + * Creates a new encoder with a smallest copy size. + * + * @param smallest Sets the smallest copy allowed. + */ + public DEZEncoder(int smallest) { + this.smallest = smallest; + } + + public void init(int sourceSize, int targetSize) { + try { + int flags = 0; + + patch.reset(); + patch.write(MAGIC); + if (smallest != 4) + flags |= DEZ_SMALLEST; + patch.write(flags); + if ((flags & DEZ_SMALLEST) != 0) + encodeInt(smallest); + encodeInt(sourceSize); + encodeInt(targetSize); + header = patch.size(); + + this.sourceSize = sourceSize; + this.targetSize = targetSize; + matchNext = 0; + recentNext = 0; + Arrays.fill(matchAddr, 0); + Arrays.fill(recentAddr, 0); + } catch (IOException ex) { + } + } + + /** + * Encodes an integer. + *

+ * Format is big-endian order encoded as: + *

+ * CXXXXXXX+ + *

+ * Where C is the continue bit. + * + * @param addr + */ + private void encodeInt(int addr) { + //int shift = ((31 - Integer.numberOfLeadingZeros(addr)) / 7) * 7; + int shift = ((32 + 6 - Integer.numberOfLeadingZeros(addr)) / 7) * 7 - 7; + + // Don't need to mask the shift since only 8 bits are significant for write() + while (shift > 0) { + int v = (addr >>> shift) | 0x80; + patch.write(v); + shift -= 7; + } + patch.write((addr & 0x7f)); + } + + /** + * Always encodes at least two bytes even for 7 bit data. + * + * @param addr + */ + private void encodeShortInt(int addr) { + //int shift = ((31 - Integer.numberOfLeadingZeros(addr | 0x2000)) / 7) * 7; + int shift = ((32 + 6 - Integer.numberOfLeadingZeros(addr | 0x2000)) / 7) * 7 - 7; + + while (shift > 0) { + int v = (addr >>> shift) | 0x80; + + patch.write(v); + shift -= 7; + } + patch.write((addr & 0x7f)); + } + + private int encLength(int addr) { + return addr != 0 ? (32 + 6 - Integer.numberOfLeadingZeros(addr)) / 7 : 1; + } + + /** + * Updates the recent address table(s). + * + * @param addr + */ + private void updateAddr(int addr) { + recentAddr[recentNext++] = addr; + recentNext &= recentAddr.length - 1; + matchAddr[matchNext++] = addr; + matchNext &= matchAddr.length - 1; + } + + /** + * Determines the most compact encoding for an address and writes it to the delta. + * + * @param addr + */ + private void encodeAddr(int addr) { + // Check for perfect match + for (int i = 0; i < matchAddr.length; i++) + if (matchAddr[i] == addr) { + patch.write(i); + updateAddr(addr); + return; + } + + // Check for near delta + int best = addr; + int d = Integer.MAX_VALUE; + int besti = 0; + for (int i = 0; i < recentAddr.length; i++) { + int r = recentAddr[i]; + int rd = Math.abs(r - addr); + if (rd < d) { + d = rd; + best = r; + besti = i; + } + } + + // shouldn't this be +1 ? it's smaller without it + if (encLength(d) < encLength(addr | 0x3fff)) { + // a delta reference is smaller + int sign = (addr - best) < 0 ? 0x20 : 0; + + patch.write(0x40 | sign | besti); + encodeInt(d); + } else { + encodeShortInt(addr); + } + + updateAddr(addr); + } + + /** + * Flushes any pending operation. + */ + private void flush() { + // 1NNNNNNN + // 0 ... 99 copy smallest ... 99+smallest + // 100 ... 123 add 1 ... 24 + // 124 copy + length (+100+smallest) + // 125 add + length (+24+1) + // 126 run + length (+3) + byte + // 127 ext command + if (last == OP_COPY) { + if (lastLen < OP_SINGLE_SPLIT + smallest) { + patch.write(0x80 | (lastLen - smallest)); + if (dump) + System.out.printf("%02x %08x: COPY %d @ $%08x\n", 0x80 | (lastLen - smallest), here, lastLen, lastAddr); + } else { + patch.write(0x80 | OP_COPY); + encodeInt(lastLen - smallest - OP_SINGLE_SPLIT); + if (dump) + System.out.printf("%02x %08x: COPY %d @ $%08x\n", 0x80 | OP_COPY, here, lastLen, lastAddr); + } + encodeAddr(lastAddr); + here += lastLen; + } else if (last == OP_ADD) { + if ((lastLen + OP_SINGLE_SPLIT - 1) < OP_SINGLE_LIMIT) { + patch.write(0x80 | (lastLen + OP_SINGLE_SPLIT - 1)); + if (dump) + System.out.printf("%02x %08x: ADD %d\n", 0x80 | (lastLen + OP_SINGLE_SPLIT - 1), here, lastLen); + } else { + patch.write(0x80 | OP_ADD); + encodeInt(lastLen - (OP_SINGLE_LIMIT - OP_SINGLE_SPLIT) - 1); + if (dump) + System.out.printf("%02x %08x: ADD %d\n", 0x80 | OP_ADD, here, lastLen); + } + patch.write(lastData, lastOff, lastLen); + here += lastLen; + } + last = -1; + } + + public void copy(int addr, int len) { + if (len < smallest) + throw new IndexOutOfBoundsException("Copy length too small"); + + if (last == OP_ADD && lastLen < 1 + 8 && len < smallest + 8) { + // 00AAACCC + if (dump) + System.out.printf("%02x %08x: ADD %d COPY %d @ $%08x\n", + ((lastLen - 1) << 3) | (len - smallest), + here, + lastLen, + len, addr); + + patch.write(((lastLen - 1) << 3) | (len - smallest)); + patch.write(lastData, lastOff, lastLen); + encodeAddr(addr); + + here += lastLen + len; + last = -1; + } else if (last == OP_COPY && lastLen < smallest + 8 && len < smallest + 8) { + // 01cccCCC + patch.write(((lastLen - smallest) << 3) | (len - smallest) | 0x40); + encodeAddr(lastAddr); + encodeAddr(addr); + + if (dump) + System.out.printf("%02x %08x: COPY %d @ $%08x COPY %d @ $%08x\n", + ((lastLen - smallest) << 3) | (len - smallest) | 0x40, + here, + lastLen, lastAddr, len, addr); + here += lastLen + len; + last = -1; + } else { + flush(); + last = OP_COPY; + lastLen = len; + lastAddr = addr; + } + } + + public void add(byte[] data, int off, int len) { + flush(); + + /* + * Let someone else handle this next call. + */ + last = OP_ADD; + lastData = data; + lastOff = off; + lastLen = len; + } + + public void run(byte b, int len) { + flush(); + + /* + * Runs are always written alone. + */ + patch.write(0x80 | OP_RUN); + encodeInt(len - 3); + patch.write(b); + + if (dump) + System.out.printf("%02x %08x: RUN $%02x %d\n", + 0x80 | OP_RUN, + here, + b & 0xff, + len); + + here += len; + } + + public byte[] toPatch() { + flush(); + + if (here != targetSize) + throw new RuntimeException("Insufficiant data"); // FIXME: better exception + + return patch.toByteArray(); + } +} diff --git a/src/notzed.dez/classes/au/notzed/dez/DEZFormat.java b/src/notzed.dez/classes/au/notzed/dez/DEZFormat.java new file mode 100644 index 0000000..56f6df8 --- /dev/null +++ b/src/notzed.dez/classes/au/notzed/dez/DEZFormat.java @@ -0,0 +1,186 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + +/** + * Defines constants used in the DEZ1 format. + *

+ *

Header

+ * 
+ *  magic: 'D' 'E' 'Z' '1'
+ *  flags: one byte
+ *         00000001               Includes 'smallest' value setting
+ *    [smallest: one integer]     Indicates the smallest value in any copy.  The default is 4.
+ *  source size: one integer
+ *  target size: one integer
+ *  instructions follow directly
+ *  ?? no epilogue defined ??
+ *
+ *

Instruction stream

+ * 
+ * Dual commands:
+ *
+ * 00lllccc dddddddd* aaaaaaaa* - add 1-8 then copy (0-7)+smallest
+ * 01cccCCC aaaaaaaa* aaaaaaaa* - copy (0-7)+smallest then copy (0-7)+smallest
+ *
+ * Single commands:
+ *
+ * 1nnnnnnn
+ *
+ *  n is interpreted as a number of ranges, inclusive:
+ * 000 ... 099        aaaaaaaa*           - copy (0-99)+smallest
+ * 100 ... 123        dddddddd*           - add 1-24
+ *         124        Ciiiiiii* aaaaaaaa* - copy i + 100 + smallest
+ *         125        Ciiiiiii* dddddddd* - add i+24+1
+ *         126        Ciiiiiii* dddddddd  - run length of i+3
+ *         127                            - reserved
+ *
+ *

+ * Integers are encoded as a compacted big-endian sequence + * with 7 bits per byte. Leading zero septets are discarded. + * The MSB of each byte is a continue bit which indicates + * another 7 bits are to be read. + *

+ *

Addresses

+ *

+ * All addresses (aaaaaaaa*) above are encoded using a rolling lookup table. + * The first byte of each address indicates whether a lookup-table specific + * address is used or an absolute address. + * 

+ *  00nnnnnn           - use address 'n' exactly.
+ *  01Smmmmm iiiiiiii* - use address 'm' plus or minus 'i'. S={ 0: plus, 1: minus }
+ *  1iiiiiii Ciiiiiii* - use address 'i' absolute.  Always at least 2 bytes.
+ *
+ *

+ * Given that 7-bit absolute addresses will rarely occur in typical data all + * absolute addresses use at least 2x bytes. This allows 2+ byte values to be encoded + * as compactly as they would otherwise be and leaves the lower 128 values for other + * encoding options. + *

+ * The address table is 64 elements long but the relative instruction can only + * reference the most recent 32 elements. It is updated each time an address is + * encoded or decoded in a simple rolling manner. + *

+ * Conceptually it can be broken into two separate parts: + * 

+ *   matchAddr[(matchNext++) & 63] = addr;
+ *   recentAddr[(recentNext++) & 31] = addr;
+ *
+ * First the encoder searches the match table; if it finds a match the address + * is encoded implicitly in the opcode via an index. + *

+ * Next the encoder finds the nearest match from the recent table as absolute + * distance. The encoder may encode the address as a relative offset from + * a member of the table if the total is smaller than encoding the address absolutely. + *

+ */ +public interface DEZFormat { + + /** + * File magic 'DEZ1'. + */ + public static final byte[] MAGIC = {'D', 'E', 'Z', '1'}; + + /** + * Header flags indicating the smallest copy size is present in the header. + */ + public static final int DEZ_SMALLEST = 0x01; + + /** + * COPY select bit in dual opcode. + */ + public static final int OP_DUAL_COPY = 0x40; + /** + * Single-operation opcode selector bit. + */ + public static final int OP_SINGLE = 0x80; + /** + * The split-point between copy and add for immediate length single operations. + *

+ * A code below this value is a COPY, otherwise it is an ADD. + */ + public static final int OP_SINGLE_SPLIT = 100; + /** + * The limit of immediate length single operations. + *

+ * This is the first non-immediate opcode (i.e. op_copy) + */ + public static final int OP_SINGLE_LIMIT = 124; + /** + * Extended COPY opcode. + */ + public static final int OP_COPY = 124; + /** + * Extended ADD opcode. + */ + public static final int OP_ADD = 125; + /** + * RUN opcode. + */ + public static final int OP_RUN = 126; + /** + * Extended command. Reserved. + */ + public static final int OP_EXT = 127; + + /** + * Indicates whether this is an absolute or indirect address. + *

+ * If this bit it set this represents the first byte of a + * multi-byte encoded integer whose value is the absolute address reference. + *

+ * If this bit is clear then the address is an indirect address. + */ + public static final int ADDR_ABSOLUTE = 0x80; + /** + * Indicates whether the address is relative or exact indirect. + *

+ * If set then the lower 5 bits select the source address slot and the + * ADDR_SIGN together with the following integer indicate the offset. + *

+ * If not set this is an exact indirect address and the lower 6 bits select the source address slot. + */ + public static final int ADDR_RELATIVE = 0x40; + /** + * The sign of the offset for a relative indirect address. + *

+ * If set the offset should be negated. + */ + public static final int ADDR_SIGN = 0x20; + + /** + * Creates a DEZ-1 format patch from byte sources. + * + * @param source + * @param target + * @return + */ + public static byte[] encode(byte[] source, byte[] target) { + return ByteDeltaEncoder.toDiff(new ByteMatcherHash(6, 4, source, 1, target), new DEZEncoder()); + } + + /** + * Applies a DEZ-1 patch. + * + * @param source + * @param patch + * @return + */ + public static byte[] decode(byte[] source, byte[] patch) { + return new au.notzed.dez.DEZDecoder(source, patch).decode(); + } +} diff --git a/src/notzed.dez/classes/module-info.java b/src/notzed.dez/classes/module-info.java new file mode 100644 index 0000000..d5b1b64 --- /dev/null +++ b/src/notzed.dez/classes/module-info.java @@ -0,0 +1,19 @@ +/* + * Copyright (C) 2019 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +module notzed.dez { + exports au.notzed.dez; +} diff --git a/src/notzed.dez/tests/au/notzed/dez/ByteMatcherHashTest.java b/src/notzed.dez/tests/au/notzed/dez/ByteMatcherHashTest.java new file mode 100644 index 0000000..858b9c3 --- /dev/null +++ b/src/notzed.dez/tests/au/notzed/dez/ByteMatcherHashTest.java @@ -0,0 +1,277 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import static org.junit.Assert.*; + +/** + * + */ +public class ByteMatcherHashTest { + + public ByteMatcherHashTest() { + } + + @BeforeClass + public static void setUpClass() { + } + + @AfterClass + public static void tearDownClass() { + } + + @Before + public void setUp() { + } + + @After + public void tearDown() { + } + + @Test + public void testGetSource() { + System.out.println("getSource"); + byte[] source = new byte[0]; + byte[] target = new byte[0]; + ByteMatcherHash instance = new ByteMatcherHash(4, 4, source, 1, target); + byte[] expResult = source; + byte[] result = instance.getSource(); + assertArrayEquals(expResult, result); + } + + @Test + public void testGetTarget() { + System.out.println("getTarget"); + byte[] source = new byte[0]; + byte[] target = new byte[0]; + ByteMatcherHash instance = new ByteMatcherHash(4, 4, source, 1, target); + byte[] expResult = target; + byte[] result = instance.getTarget(); + assertArrayEquals(expResult, result); + } + + @Test + public void testEmpty() { + System.out.println("testEmpty"); + + byte[] source = new byte[0]; + byte[] target = new byte[0]; + ByteMatcherHash bm = new ByteMatcherHash(4, 4, source, 1, target); + + assertEquals(bm.nextMatch(), ByteMatcherHash.EOF); + } + + @Test + public void testRun() { + System.out.println("testRun"); + + byte[] source = new byte[0]; + byte[] target = "aaa".getBytes(); + ByteMatcherHash bm = new ByteMatcherHash(2, 4, source, 1, target); + + assertEquals(ByteMatcherHash.RUN, bm.nextMatch()); + assertEquals(3, bm.getLength()); + assertEquals('a', bm.getRunByte()); + + assertEquals(ByteMatcherHash.EOF, bm.nextMatch()); + } + + @Test + public void testRunEnd() { + System.out.println("testRunEnd"); + + byte[] source = new byte[0]; + byte[] target = "abcdaaa".getBytes(); + ByteMatcherHash bm = new ByteMatcherHash(2, 4, source, 1, target); + + assertEquals(ByteMatcherHash.RUN, bm.nextMatch()); + assertEquals(3, bm.getLength()); + assertEquals('a', bm.getRunByte()); + assertEquals(4, bm.getTargetOffset()); + + assertEquals(ByteMatcherHash.EOF, bm.nextMatch()); + } + + @Test + public void testCopy() { + System.out.println("testCopy"); + + byte[] source = "abcdefg".getBytes(); + byte[] target = "abcdefg".getBytes(); + ByteMatcherHash bm = new ByteMatcherHash(4, 4, source, 1, target); + + assertEquals(ByteMatcherHash.COPY, bm.nextMatch()); + assertEquals(source.length, bm.getLength()); + assertEquals(0, bm.getMatchOffset()); + assertEquals(source, bm.getBlockArray(bm.getMatchOffset())); + assertEquals(0, bm.getBlockOffset(bm.getMatchOffset())); + assertEquals(ByteMatcherHash.EOF, bm.nextMatch()); + } + + @Test + public void testCopyEnd() { + System.out.println("testCopyEnd"); + + byte[] source = "xxxxabcd".getBytes(); + byte[] target = "abcdabcd".getBytes(); + ByteMatcherHash bm = new ByteMatcherHash(4, 4, source, 1, target); + + assertEquals(ByteMatcherHash.COPY, bm.nextMatch()); + assertEquals(4, bm.getLength()); + assertEquals(4, bm.getMatchOffset()); + assertEquals(source, bm.getBlockArray(bm.getMatchOffset())); + assertEquals(4, bm.getBlockOffset(bm.getMatchOffset())); + + assertEquals(ByteMatcherHash.COPY, bm.nextMatch()); + assertEquals(4, bm.getLength()); + + assertEquals(ByteMatcherHash.EOF, bm.nextMatch()); + } + + @Test + public void testCopyTarget() { + System.out.println("testCopyTarget"); + + byte[] source = "wxyz".getBytes(); + byte[] target = "abcdabcd".getBytes(); + ByteMatcherHash bm = new ByteMatcherHash(4, 4, source, 1, target); + + assertEquals(ByteMatcherHash.COPY, bm.nextMatch()); + assertEquals(4, bm.getMatchOffset()); + assertEquals(4, bm.getTargetOffset()); + assertEquals(target, bm.getBlockArray(bm.getMatchOffset())); + + assertEquals(ByteMatcherHash.EOF, bm.nextMatch()); + } + + @Test + public void testNomatch() { + System.out.println("testNomatch"); + + byte[] source = "abcdefgh".getBytes(); + byte[] target = "ijklmnop".getBytes(); + ByteMatcherHash bm = new ByteMatcherHash(4, 4, source, 1, target); + + assertEquals(ByteMatcherHash.EOF, bm.nextMatch()); + assertEquals(0, bm.getMatchOffset()); + } + + @Test + public void testCopy2() { + System.out.println("testCopy2"); + + byte[] source = "abcdefgh".getBytes(); + byte[] target = "abcdabcd".getBytes(); + ByteMatcherHash bm = new ByteMatcherHash(4, 4, source, 1, target); + + assertEquals(ByteMatcherHash.COPY, bm.nextMatch()); + assertEquals(4, bm.getLength()); + assertEquals(0, bm.getMatchOffset()); + assertEquals(0, bm.getTargetOffset()); + assertEquals(source, bm.getBlockArray(bm.getMatchOffset())); + assertEquals(0, bm.getBlockOffset(bm.getMatchOffset())); + + assertEquals(ByteMatcherHash.COPY, bm.nextMatch()); + assertEquals(4, bm.getLength()); + assertEquals(4, bm.getTargetOffset()); + + assertEquals(ByteMatcherHash.EOF, bm.nextMatch()); + } + + static class Expected { + + int state; + int length; + int targetOffset; + int matchOffset; // or match byte + + public Expected(int state, int length, int targetOffset, int matchOffset) { + this.state = state; + this.length = length; + this.targetOffset = targetOffset; + this.matchOffset = matchOffset; + } + + void check(ByteMatcherHash bm, int state) { + } + + static void check(Expected[] list, ByteMatcherHash bm) { + for (int i = 0; i < list.length; i++) { + Expected e = list[i]; + int state = bm.nextMatch(); + assertEquals(e.state, state); + assertEquals(e.length, bm.getLength()); + assertEquals(e.targetOffset, bm.getTargetOffset()); + if (state == ByteMatcher.COPY) { + assertEquals(e.matchOffset, bm.getMatchOffset()); + } else if (state == ByteMatcher.RUN) { + assertEquals(e.matchOffset, bm.getRunByte() & 0xff); + } + } + + assertEquals(bm.nextMatch(), ByteMatcher.EOF); + } + + static void make(String name, ByteMatcherHash bm) { + int s; + + System.out.printf("static Expected[] %s = {\n", name); + while ((s = bm.nextMatch()) != bm.EOF) { + switch (s) { + case ByteMatcherHash.COPY: + System.out.printf("new Expected(ByteMatcher.COPY, %d, %d, %d),\n", + bm.getLength(), + bm.getTargetOffset(), + bm.getMatchOffset()); + break; + case ByteMatcherHash.RUN: + System.out.printf("new Expected(ByteMatcher.RUN, %d, %d, 0x%02x),\n", + bm.getLength(), + bm.getTargetOffset(), + bm.getRunByte() & 0xff); + break; + } + } + System.out.printf("};\n"); + } + + } + static Expected[] testSequence = { + new Expected(ByteMatcher.COPY, 5, 0, 39), + new Expected(ByteMatcher.COPY, 7, 17, 57), + new Expected(ByteMatcher.COPY, 4, 24, 0), + new Expected(ByteMatcher.COPY, 5, 29, 4) + }; + + @Test + public void testSequence() { + System.out.println("testSequence"); + + byte[] source = "the rains in spain falls mainly on the plains.".getBytes(); + byte[] target = "plain bread melts melts the brains.".getBytes(); + ByteMatcherHash bm = new ByteMatcherHash(4, 4, source, 1, target); + + //Expected.make("testSequence", bm); + Expected.check(testSequence, bm); + } + +} diff --git a/src/notzed.dez/tests/au/notzed/dez/DEZEncoderTest.java b/src/notzed.dez/tests/au/notzed/dez/DEZEncoderTest.java new file mode 100644 index 0000000..4617698 --- /dev/null +++ b/src/notzed.dez/tests/au/notzed/dez/DEZEncoderTest.java @@ -0,0 +1,348 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import static org.junit.Assert.*; + +/** + * There could be plenty more tests here for all address and instruction variations. + */ +public class DEZEncoderTest { + + public DEZEncoderTest() { + } + + @BeforeClass + public static void setUpClass() { + } + + @AfterClass + public static void tearDownClass() { + } + + @Before + public void setUp() { + } + + @After + public void tearDown() { + } + + @Test + public void testCopy() { + System.out.println("copy"); + int addr = 0; + int len = 5; + DEZEncoder instance = new DEZEncoder(); + + instance.init(len, len); + instance.copy(addr, len); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(patch, new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + (byte) len, + (byte) len, + (byte) (128 + len - 4), 0 + }); + } + + @Test + public void testAdd() { + System.out.println("add"); + byte[] data = {1, 2, 3, 4}; + int off = 0; + int len = data.length; + DEZEncoder instance = new DEZEncoder(); + + instance.init(0, len); + instance.add(data, off, len); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(patch, new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 0, + (byte) len, + (byte) (128 + 100 + len - 1), 1, 2, 3, 4 + }); + } + + @Test + public void testRun() { + System.out.println("run"); + byte b = 65; + int len = 5; + DEZEncoder instance = new DEZEncoder(); + + instance.init(0, len); + instance.run(b, len); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(patch, new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 0, + (byte) len, + (byte) (126 | 0x80), + (byte) (len - 3), + b + }); + } + + @Test + public void testRun1Byte() { + System.out.println("run 1 byte"); + byte b = 65; + int len = 127 + 3; + DEZEncoder instance = new DEZEncoder(); + + instance.init(0, len); + instance.run(b, len); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(patch, new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 0, + (byte) ((len >> 7) | 0x80), + (byte) (len & 127), + (byte) (126 | 0x80), + (byte) (len - 3), + b + }); + } + + @Test + public void testRun2Byte() { + System.out.println("run 2 byte"); + byte b = 65; + int len = 127 + 3 + 1; + DEZEncoder instance = new DEZEncoder(); + + instance.init(0, len); + instance.run(b, len); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(patch, new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 0, + (byte) ((len >> 7) | 0x80), + (byte) (len & 127), + (byte) (126 | 0x80), + (byte) (((len - 3) >> 7) | 0x80), + (byte) ((len - 3) & 0x7f), + b + }); + } + + @Test + public void testAddAdd() { + System.out.println("add add"); + byte[] data = {1, 2, 3, 4, 5, 6, 7, 8}; + int len = data.length; + DEZEncoder instance = new DEZEncoder(); + + instance.init(0, len); + instance.add(data, 0, 5); + instance.add(data, 5, 3); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(patch, new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 0, + (byte) len, + (byte) (128 + 100 + 5 - 1), 1, 2, 3, 4, 5, + (byte) (128 + 100 + 3 - 1), 6, 7, 8 + }); + } + + @Test + public void testCopyCopyShort() { + System.out.println("copy copy short"); + int len = 20; + DEZEncoder instance = new DEZEncoder(); + int smallest = 4; + + instance.init(16, len); + instance.copy(0, 10); + instance.copy(0, 10); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 16, + (byte) len, + (byte) ((10 - smallest) << 3 | (10 - smallest) | 0x40), + 0, 0 + }, patch); + } + + @Test + public void testCopyCopyShortSmallest() { + System.out.println("copy copy short smallest!=4"); + int len = 20; + int smallest = 7; + DEZEncoder instance = new DEZEncoder(smallest); + + instance.init(16, len); + instance.copy(0, 10); + instance.copy(0, 10); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + DEZEncoder.DEZ_SMALLEST, + (byte) smallest, + 16, + (byte) len, + (byte) ((10 - smallest) << 3 | (10 - smallest) | 0x40), + 0, 0 + }, patch); + } + + private void dump(byte[] patch) { + for (int i = 0; i < patch.length; i++) + System.out.printf("%2d: %02x\n", i, patch[i] & 0xff); + } + + @Test + public void testCopyAddrMinus() { + System.out.println("copy addr -"); + int len = 20; + int smallest = 4; + DEZEncoder instance = new DEZEncoder(); + + instance.init(16, len); + instance.copy(10, 10); + instance.copy(8, 10); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 16, + (byte) len, + (byte) (((10 - smallest) << 3) | (10 - smallest) | 0x40), + 0x40, 10, + // - 2 + 0x60, 2 + }, patch); + } + + @Test + public void testCopyAddrPlus() { + System.out.println("copy addr +"); + int len = 20; + int smallest = 4; + DEZEncoder instance = new DEZEncoder(); + + instance.init(16, len); + instance.copy(8, 10); + instance.copy(10, 10); + + byte[] patch = instance.toPatch(); + + assertArrayEquals(new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 16, + (byte) len, + (byte) (((10 - smallest) << 3) | (10 - smallest) | 0x40), + 0x40, 8, + // + 2 + 0x40, 2 + }, patch); + } + + @Test + public void testFormatEmpty() { + System.out.println("fmt empty"); + + DEZEncoder instance = new DEZEncoder(); + byte[] source = new byte[0]; + byte[] target = new byte[0]; + instance.init(source.length, target.length); + + byte[] patch = instance.toPatch(); + + assertEquals(7, patch.length); + assertArrayEquals(patch, new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 0, + 0}); + } + + @Test + public void testFormatSmallest() { + System.out.println("smallest non-default"); + + DEZEncoder instance = new DEZEncoder(6); + byte[] source = new byte[0]; + byte[] target = new byte[0]; + instance.init(source.length, target.length); + + byte[] patch = instance.toPatch(); + + assertEquals(8, patch.length); + assertArrayEquals(patch, new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + DEZEncoder.DEZ_SMALLEST, + 6, + 0, + 0}); + } + + @Test + public void testFormatSmallestDefault() { + System.out.println("smallest default=4"); + + DEZEncoder instance = new DEZEncoder(4); + byte[] source = new byte[0]; + byte[] target = new byte[0]; + instance.init(source.length, target.length); + + byte[] patch = instance.toPatch(); + + assertEquals(7, patch.length); + assertArrayEquals(patch, new byte[]{ + DEZEncoder.MAGIC[0], DEZEncoder.MAGIC[1], DEZEncoder.MAGIC[2], DEZEncoder.MAGIC[3], + 0, + 0, + 0}); + } + +} diff --git a/src/notzed.dez/tests/au/notzed/dez/DEZIT.java b/src/notzed.dez/tests/au/notzed/dez/DEZIT.java new file mode 100644 index 0000000..9e683cc --- /dev/null +++ b/src/notzed.dez/tests/au/notzed/dez/DEZIT.java @@ -0,0 +1,122 @@ +/* + * Copyright (C) 2015 Michael Zucchi + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the + * License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package au.notzed.dez; + +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import static org.junit.Assert.*; + +/** + * + */ +public class DEZIT { + + public DEZIT() { + } + + @BeforeClass + public static void setUpClass() { + } + + @AfterClass + public static void tearDownClass() { + } + + @Before + public void setUp() { + } + + @After + public void tearDown() { + } + + @Test + public void testSmallest() throws Exception { + System.out.println("decode/encode smallest changing"); + + byte[] source = "the rains in Spain fall mainly on the plains".getBytes(); + byte[] target = "plain breads in Spain aids your dames brains".getBytes(); + + for (int smallest = 4; smallest <= 16; smallest++) { + DEZEncoder de = new DEZEncoder(smallest); + ByteMatcherHash matcher = new ByteMatcherHash(4, smallest, source, 1, target); + byte[] patch = ByteDeltaEncoder.toDiff(matcher, de); + DEZDecoder dd = new DEZDecoder(source, patch); + byte[] result = dd.decode(); + + assertArrayEquals(result, target); + } + } + + @Test + public void testEmptySource() throws Exception { + System.out.println("empty source"); + + byte[] source = "".getBytes(); + byte[] target = "plain breads in Spain aids your dames brains".getBytes(); + + for (int smallest = 4; smallest <= 16; smallest++) { + DEZEncoder de = new DEZEncoder(smallest); + ByteMatcherHash matcher = new ByteMatcherHash(4, smallest, source, 1, target); + byte[] patch = ByteDeltaEncoder.toDiff(matcher, de); + DEZDecoder dd = new DEZDecoder(source, patch); + byte[] result = dd.decode(); + + assertArrayEquals(result, target); + } + } + + @Test + public void testEmptyTarget() throws Exception { + System.out.println("empty target"); + + byte[] source = "the rains in Spain fall mainly on the plains".getBytes(); + byte[] target = "".getBytes(); + + for (int smallest = 4; smallest <= 16; smallest++) { + DEZEncoder de = new DEZEncoder(smallest); + ByteMatcherHash matcher = new ByteMatcherHash(4, smallest, source, 1, target); + byte[] patch = ByteDeltaEncoder.toDiff(matcher, de); + DEZDecoder dd = new DEZDecoder(source, patch); + byte[] result = dd.decode(); + + assertArrayEquals(result, target); + } + } + + @Test + public void testRuns() throws Exception { + System.out.println("runs"); + + byte[] source = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".getBytes(); + byte[] target = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".getBytes(); + + for (int smallest = 4; smallest <= 16; smallest++) { + DEZEncoder de = new DEZEncoder(smallest); + ByteMatcherHash matcher = new ByteMatcherHash(4, smallest, source, 1, target); + byte[] patch = ByteDeltaEncoder.toDiff(matcher, de); + DEZDecoder dd = new DEZDecoder(source, patch); + byte[] result = dd.decode(); + + assertArrayEquals(result, target); + } + } + +}