Pattern matching and defaults for structs
authorNot Zed <notzed@gmail.com>
Tue, 4 Jan 2022 22:49:15 +0000 (09:19 +1030)
committerNot Zed <notzed@gmail.com>
Tue, 4 Jan 2022 22:49:15 +0000 (09:19 +1030)
Added ffmpeg example.
Changed test-api to be c-like example.

18 files changed:
README
src/export-defines
src/export.cc
src/generate-native
src/template/Memory.java
test-api/Makefile
test-api/README [new file with mode: 0644]
test-api/api.api [new file with mode: 0644]
test-api/api.c
test-api/api.h
test-api/src/api/test/TestAPI.java
test-ffmpeg/Makefile [new file with mode: 0644]
test-ffmpeg/README [new file with mode: 0644]
test-ffmpeg/ffmpeg.api [new file with mode: 0644]
test-ffmpeg/ffmpeg.h [new file with mode: 0644]
test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java [new file with mode: 0644]
test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java [new file with mode: 0644]
test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java [new file with mode: 0644]

diff --git a/README b/README
index 6fe7c06..cdb34e6 100644 (file)
--- a/README
+++ b/README
@@ -2,91 +2,53 @@
 Introduction
 ------------
 
-This is an alternative to jextract for the OpenJDK panama project
-which in turn is a project to implement Java to Native interfaces at
-runtime.
+This is some experiments with the panama-foreign abi JEP for invoking
+native C functions from Java directly and without JNI.
 
-It uses a gcc plugin for type extraction and a "simple" perl script to
-transform this into Java source files with the appropriate annotations
-for panama to do it's thing.
+The main goal is to experiment with creating a "java friendly" and
+mostly type-safe api directly in one step, without requiring
+additional wrapping or inconvenient use.
 
-Compile
--------
-
-Edit the makefile to point to your gcc directory and make.
-
-$ cd src
-$ vi Makefile
-$ make
-
-generate is a perl script.
-
-See test-api/* for a simple example.  JAVA_HOME must point to
-a compatible panama-enabled jdk.
-
-Use dump.so
------------
-
-dump.so is a gcc plugin which captures structures, unions, enums,
-exported functions, and a few typedefs into a perl source file
-containing a single hash.
-
-Put all the include files required for your api in api.h.
-
-$ gcc -fplugin=src/export.so -fplugin-arg-export-output=api.pm api.h -o /dev/null
+It's roughly tracking the openjdk repository of the same date.
 
-api.pm is a text file of perl source-code which contains a descriptor
-of all supported types found.
+It uses a gcc plugin to compile the headers to obtain most of the api
+information, but requires cpp and perl to extract the values of
+#define constants.
 
-Use generate
-------------
+This api information is then converted to Java wrappers using a
+config-directed perl script `generate-native'.  The config is flexible
+enough to either generate c-like libraries of static functions or
+object-oriented layouts with data, static and object functions grouped
+by class.
 
-Quick and dirty:
+test-vulkan uses a different approach as the vulkan api is officially
+defined by an xml registry and not the generated C headers.  This is
+directly converted to about-as-java-friendly a vulkan api as one can
+hope for (all those structs you need to initialise get messy in any
+language).  Most of the script is converting the registry vk.xml file
+into a usable structure as it's really only designed for outputting c
+headers.
 
-$ generate -d dir -t name.space -c APILib -lapi ./api.pm
+Compile
+-------
 
-Generate all structures in api.pm (the ./ is required) and place all
-functions into dir/name/space/APILib.java which will link with
-libapi.so.
+Requirements are gcc and cpp, perl, GNU make.
 
-The generator is quite flexible, sub-sets of functions can be placed
-in alternative *Lib.java files, types can be filtered.  If a function
-or type is specified explicitly, then only those types it requires are
-included in the generated output (recursively).
+First run make in src.
 
-See the top of the file for details, it's probably up to date.
+See test-*/* for examples.  JAVA_HOME must point to a compatible
+panama-enabled jdk.
 
 Status
 ------
 
-This is currently just a 'quick and dirty' weekend hack.  The tools
-spew a lot of debug.  dump.cc is basic c-compatible-c++, because I
-don't know c++.
-
-dump.so works by hooking into the gcc pre-compiled headers function,
-it might work with a c source file as well.  It's only been used with
-gcc 9.
-
-The dumper will hard-fail if it sees defintions it doesn't understand,
-and currently it's only been tested with a small nubmer of libraries.
-
-Enumerations are currently translated to integers.  The generator
-doesn't currently export any of the symbolic values to Java but it is
-easy to add.
-
-varags is not implemented.
-
-All function pointers are translated to a unique set of functional
-interfaces whose names are based on the function signature.  So the
-type names are ugly but they are stable.
-
-Anonymous type names are based on the structure member they belong to
-so they may not be unique.
-
-It will only ever support c.
+It's all work in progress of course.
 
-Only tested against the panama 'openjdk-14-panama+1-15_linux' on
-slackware-current @ 2020.01.01 and gcc 9.2.0.
+* bitfields not implemented yet.
+* varargs is not implemented.
+* the support library in Memory.java is copied to each output api, but should
+  be global.
+* the scope and object lifecycle stuff is not really sorted out yet.
 
 License
 -------
@@ -99,3 +61,4 @@ Links
 
  * https://www.zedzone/software/panamaz.html - project page.
  * https://openjdk.java.net/projects/panama - openjdk panama page.
+ * https://github.com/openjdk/panama-foreign - openjdk panama source.
index ece2b0e..79b35e5 100755 (executable)
 #!/usr/bin/perl
 
-# the gcc plugin doesn't have access to the #defines, so this is a 'simple' way to get them.
+use Data::Dumper;
 
-@matchDefine = ();
-@matchInclude = ();
-
-@includes = ();
+my @includes = ();
+my $header;
+my $control = "api-defines.def";
+my $output;
 
 while (@ARGV) {
-    my $cmd = shift(@ARGV);
-
-    if ($cmd eq "-d") {
-       my $v = shift(@ARGV);
-       push @matchDefine, $v;
-    } elsif ($cmd eq "--define-file") {
-       my $file = shift(@ARGV);
-       push @matchDefine, readMatchFile($file);
-    } elsif ($cmd eq "-i") {
-       my $v = shift(@ARGV);
-       push @matchInclude, $v;
-    } elsif ($cmd eq "-I") {
-       my $v = shift(@ARGV);
-       push @includes, $v;
-    } elsif ($cmd =~ "-I(.*)") {
-       push @includes, $1;
-    } else {
-       $header = $cmd;
-    }
+       my $cmd = shift;
+
+       if ($cmd eq "-t") {
+               $package = shift;
+    } elsif ($cmd eq "-d") {
+               $output = shift;
+    } elsif ($cmd eq "-v") {
+               $verbose++;
+       } elsif ($cmd eq "-I") {
+               push @includes, shift;
+       } else {
+               $control = $cmd;
+       }
+}
+
+die ("no output specified") if !$output;
+
+my $defs = loadControlFile($control);
+my @exports = @{$defs->{define}};
+my %rawDefines = (); # indexed by header
+
+my $CPPFLAGS = "";
+
+foreach $i (@includes) {
+       $CPPFLAGS .= " '-I$i'";
+}
+
+# flatten the generic format to a more usable format, work out defaults (. and syntax check?)
+foreach $export (@exports) {
+       my $includes = 0, $excludes = 0;
+
+       foreach $inc (@{$export->{items}}) {
+               my @options = @{$inc->{options}};
+
+               if ($inc->{match} =~ m@^/(.*)/$@) {
+                       $inc->{regex} = qr/$1/;
+               } else {
+                       $inc->{regex} = qr/^$inc->{match}$/;
+               }
+
+               $inc->{mode} = "include";
+               foreach $o (@{$inc->{options}}) {
+                       if ($o =~ m/^(exclude|include|file-include|file-exclude)$/) {
+                               $inc->{mode} = $o;
+                       }
+               }
+               if ($inc->{mode} =~ m/include/) {
+                       $includes += 1;
+               } else {
+                       $excludes += 1;
+               }
+       }
+
+       $export->{default} = 'all';
+       $export->{default} = 'none' if ($includes > 0);
+
+       my @options = @{$export->{options}};
+
+       foreach $o (@options) {
+               if ($o =~ m/header=(.*)/) {
+                       $export->{header} = $1;
+               } elsif ($o =~ m/default=(.*)/) {
+                       $export->{default} = $1;
+               } elsif ($#options == 0) {
+                       $export->{header} = $o;
+               } else {
+                       print STDERR "unknown defines option '$o'\n";
+               }
+       }
+
+       # insert ignore <built-in> thing
+       if ($export->{default} eq 'all') {
+               unshift @{$export->{items}}, { regex => qr/<built-in>/, mode => 'file-exclude' };
+       }
+
+       die ("no header for '$export->{name}'") if !$export->{header};
+}
+
+# load all defines once and link in
+# keep_comments is a bit broken
+foreach $export (@exports) {
+       my $header = $export->{header};
+
+       if (!defined($rawDefines{$header})) {
+               $rawDefines{$header} = scanDefines($header, { CPPFLAGS=>$cppflags, keep_comments=>0 });
+       }
+
+       $export->{rawDefines} = $rawDefines{$header};
+}
+
+foreach $export (@exports) {
+       # extract matching #defines
+       my @defines = ();
+
+       foreach $d (@{$export->{rawDefines}}) {
+               my $output = 0;
+               my $matches = 0;
+
+               print "? $d->{name} $d->{file}" if $verbose;
+               foreach $inc (@{$export->{items}}) {
+                       if ($inc->{mode} eq "include") {
+                               $matches = $d->{name} =~ m/$inc->{regex}/;
+                               $output = 1 if $matches;
+                       } elsif ($inc->{mode} eq "exclude") {
+                               $matches = $d->{name} =~ m/$inc->{regex}/;
+                       } elsif ($inc->{mode} eq "file-include") {
+                               $matches = $d->{file} =~ m/$inc->{regex}/;
+                               $output = 1 if $matches;
+                       } elsif ($inc->{mode} eq "file-exclude") {
+                               $matches = $d->{file} =~ m/$inc->{regex}/;
+                       }
+                       print " ($inc->{mode} '$inc->{match}' =$matches)" if $verbose;
+                       last if $matches;
+               }
+
+               $output = 1 if (!$matches && $export->{default} eq "all");
+
+               print " output=$output\n" if $verbose;
+
+               push (@{$export->{export}}, $d) if $output;
+       }
+
+}
+
+open (my $fh, ">", "$output~") || die("can't open $output~ for writing");
+
+export($fh, \@exports);
+
+close ($fh) || die("error writing");
+rename "$output~",$output || die("error overwriting $output");
+
+exit 0;
+
+# ######################################################################
+
+sub uniq {
+  my %seen;
+  return grep { !$seen{$_}++ } @_;
+}
+
+sub export {
+       my $fp = shift;
+       my @exports = @{shift @_};
+
+       print $fp <<END;
+#include <stdio.h>
+#include <stdint.h>
+END
+       foreach $h (uniq map { $_->{header} } @exports) {
+               print $fp  "#include \"$h\"\n";
+       }
+
+       # this is effectively a function overloading system so that the
+       # compiler can select the datatype of the evaluation of the
+       # definition.
+       print $fp <<END;
+/* note unsigned types are output as signed for java compatability */
+#define FMT(x) \\
+       __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), float),    "value=>%a, type=>'f32'", \\
+       __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), double),   "value=>%a, type=>'f64'", \\
+       __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), int32_t),  "value=>%d, type=>'i32'", \\
+       __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), uint32_t), "value=>%d, type=>'u32'", \\
+       __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), int64_t),  "value=>%ld, type=>'i64'", \\
+       __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), uint64_t), "value=>%ld, type=>'u64'", \\
+       __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), typeof(char[])),  "value=>'%s', type=>'string'", "type => undefined" )))))))
+
+int main(int argc, char **argv) {
+ FILE *fp = fopen(argv[1], "w");
+
+ fputs("{\\n", fp);
+END
+
+       foreach $export (@exports) {
+               print $fp <<END;
+fputs("'define:$export->{name}' => { name => '$export->{name}', type => 'define', values => [\\n", fp);
+END
+               foreach $d (@{$export->{export}}) {
+                       my $docomment;
+
+                       if ($d->{comment}) {
+                               my $comment = $d->{comment};
+                               $comment =~ s@(['"])@\\\\\\$1@g;
+                               $comment =~ s@\n@\\n@g;
+                               $docomment = ", comment => '$comment' ";
+                       }
+
+                       print $fp <<END;
+ fputs("  { name => \\"$d->{name}\\", ", fp);
+ fprintf(fp, FMT($d->{name}), ($d->{name}));
+ fputs("$docomment},\\n", fp);
+END
+               }
+               print $fp <<END;
+ fputs(" ],\\n},\\n", fp);
+END
+       }
+       print $fp <<END;
+ fprintf(fp, "}\\n");
+ fclose(fp);
+}
+END
 }
 
-my $all = join ('|', @matchDefine);
+# args: filename, \@defines, \@defineList
+# export a c file generator
+sub export_generator {
+       my $filename = shift;
+       my @defines = @{shift @_};
+       my @defineList = @{shift @_};
+}
 
-$matchDefine = qr/($all)/;
+# args: header, \%options
+# options = {
+#   CPPFLAGS => 'flags'
+#   keep_comments => 1 to keep comments
+# }
+# returns a list of
+# {
+#   name=>'name',
+#   comment='comment',
+#   file='filename',
+#   line=linenumber
+# }
+sub scanDefines {
+       my $header = shift;
+       my %o = %{shift @_};
+    my $lastc = "";
+       my $source;
+       my $sourceLine;
 
-$all = join('|', @matchInclude);
+    print STDERR "Scanning $header\n";
 
-$matchInclude = qr/($all)/;
+    open (my $in,"-|","cpp -dD ".($o{keep_comments} ? '-CC' : '')." $o{CPPFLAGS} $header") || die("Can't find include file: $header");
+    while (<$in>) {
+               # line markers
+               if (m/^\# (\d+) \"([^\"]*)/) {
+                       $sourceLine = $1;
+                       $source = $2;
+                       next;
+               }
+               # accumulate comments
+               # single line comments override multi-line
+               if ($o{keep_comments}) {
+                       if (m@/\*(.*)\*/@) {
+                               do {
+                                       $lastc = $1 if $lastc eq "";
+                                       s@/\*.*\*/@@;
+                               } while (m@/\*(.*)\*/@);
+                       } elsif (m@/\*(.*)@) {
+                               my $com = "$1\n" if $1;
+                               while (<$in>) {
+                                       chop;
+                                       if (m@(.*)\*/@) {
+                                               $com .= $1 if $1;
+                                               last;
+                                       } else {
+                                               $com .= "$_\n";
+                                       }
+                               }
+                               $lastc = $com if $com && $lastc eq "";
+                       } elsif (m@//(.*)@) {
+                               $lastc = $1 if $1;
+                               s@//.*@@;
+                       }
+               }
 
-print "all: $all\n";
-print "$matchDefine\n";
+               if (m/^\s*#define\s+(\w*)\(/) {
+                       # ignore macros
+                       $lastc = "";
+               } elsif (m/^\s*#define\s+(\w*)\s*$/) {
+                       # ignore empty defines
+                       $lastc = "";
+               } elsif (m/^\s*#define\s+(\w+)/) {
+                       my $name = $1;
+                       my %define = ();
 
-sub readMatchFile {
-    my $path = shift @_;
-    my @lines = ();
+                       $define{name} = $name;
+                       $define{comment} = $lastc if $lastc ne "";
+                       $define{file} = $source;
+                       $define{line} = $sourceLine;
 
-    open(my $f,"<$path");
-    while (<$f>) {
-       chop;
-       next if m/^#/;
+                       push @defines, \%define;
 
-       push @lines, $_;
-    }
-    close($f);
+                       $lastc = "";
+               }
 
-    my $all = join ('|', @lines);
+               $sourceLine++;
+       }
+       close $in;
 
-    return qr/^($all)$/;
+       return \@defines;
 }
 
-$dump = 0;
-
-$args = join("", map { "\"-I$_\"" } @includes);
-
-open IN,"gcc -C -E -dD $args $header|";
-# only #defines that are defined in the same directory or file as header?
-while (<IN>) {
-    if (m/^# (\d*) \"(.*)\"/) {
-       $dump = ($2 =~ /$matchInclude/);
-    }
-    if ($dump) {
-       if (m/^#define\s+(\w*)\s+(.*)/) {
-           my $name = $1;
-           my $def = $2;
-
-           if ($name =~ /$matchDefine/) {
-               if ($def =~ m/L$/) {
-                   print "public static final long $name = (long)$def;\n"; if ($def =~ m/L$/);
-               } elsif ($def =~ m/^(0x)?[0-9]*$/) {
-                   print "public static final long $name = (int)$def;\n"; if ($def =~ m/L$/);
-               } elsif ($def =~ m/f$/) {
-                   print "public static final float $name = $def;\n"; if ($def =~ m/L$/);
-               } elsif ($def =~ m/^[\.E-+0-9]*$/) {
+# TODO: library
+sub loadControlFile {
+       my $path = shift @_;
+       my %def = ();
+       my $target;
+
+       open (my $d,"<",$path);
+
+       while (<$d>) {
+               next if /\s*\#/;
+
+               chop;
+
+               if ($target) {
+                       if (m/\s*\}\s*$/) {
+                               undef $target;
+                       } elsif (/^\s*(\S+)\s*(.*)/) {
+                               my @options = split(/\s+/,$2);
+                               push @{$target->{items}}, {
+                                       match => $1,
+                                       options => \@options
+                               };
+                       }
+               } elsif (/^(\w+)\s+(\S*)\s*(.*)\s+\{/) {
+                       my @options = split(/\s+/,$3);
+
+                       $target = {
+                               type => $1,
+                               name => $2,
+                               options => \@options,
+                               items => []
+                       };
+                       push @{$def{$1}}, $target;
+               } elsif (/\S/) {
+                       die("invalid line: %_");
                }
-               #if ($def =~ m/^0x/) {
-               #}
-           }
        }
-    }
+
+       close $d;
+
+       return \%def;
 }
-close IN;
index 4a34c58..a37f0bd 100644 (file)
@@ -697,7 +697,7 @@ static void export_type(tree type, const char *names) {
                                names, names, size, TYPE_UNSIGNED(target) ? 'u' : 'i', size);
 
                        for (tree v = TYPE_VALUES(target); v != NULL; v = TREE_CHAIN (v)) {
-                               generate("\t{ label => '%s', value => '%ld' },\n",
+                               generate("\t{ name => '%s', value => '%ld' },\n",
                                        IDENTIFIER_POINTER(TREE_PURPOSE(v)),
                                        tree_to_shwi(TREE_VALUE(v)));
                        }
@@ -873,7 +873,7 @@ static void export_type(tree type, const char *names) {
                        names, names, size, TYPE_UNSIGNED(type) ? 'u' : 'i', size);
 
                for (tree v = TYPE_VALUES(type); v != NULL; v = TREE_CHAIN (v)) {
-                       generate("\t{ label => '%s', value => '%ld' },\n",
+                       generate("\t{ name => '%s', value => '%ld' },\n",
                                IDENTIFIER_POINTER(TREE_PURPOSE(v)),
                                tree_to_shwi(TREE_VALUE(v)));
                }
@@ -977,7 +977,7 @@ static void plugin_finish(void *event_data, void *user_data) {
        for (struct node *n = dumped.list.head; n; n=n->next)
                generate("# %s\n", n->name);
 
-       generate(");\n");
+       generate("}\n");
        fclose(output_file);
 
        if (debug_level > 0)
@@ -1007,7 +1007,7 @@ int plugin_init(struct plugin_name_args *plugin_info, struct plugin_gcc_version
                exit(EXIT_FAILURE);
        }
 
-       generate("%%data = (\n");
+       generate("{\n");
 
        register_callback(plugin_info->base_name, PLUGIN_FINISH_DECL, plugin_finish_decl, NULL);
        register_callback(plugin_info->base_name, PLUGIN_FINISH_TYPE, plugin_finish_type, NULL);
index 55eaee7..62cf2fb 100755 (executable)
@@ -2,8 +2,14 @@
 
 # -*- Mode:perl; perl-indent-level:4;tab-width:4; -*-
 
-# TODO: get/set should take a typed field for struct/union *
-# TODO: a flag for func_name to funcName option
+# TODO: global defaults and/or pattern matching on api targets, partially done
+# TODO: perhaps an option which just dumps everything it finds at some level
+# TODO: the code here isn't a complete mess but could be improved
+#       - parameterise some functions
+# TODO: map int to boolean where appropriate
+# TODO: arrays with specified lengths passed as arguments could be size-checked in function stubs.
+# TODO: error codes -> exceptions?
+# TODO: auto-loading of libraries (library xx load=blah) option.
 
 use Data::Dumper;
 use File::Basename;
@@ -14,11 +20,13 @@ $scriptPath = dirname(__FILE__);
 $package = "";
 $output = "bin";
 $verbose = 0;
+$apidef = "api.def";
 
 # usage
 #  -t package    target package
 #  -d directory  output root
 #  -v            verbose
+#  -a datafile   add datafile to the dataset, can be from export.so or export-defines, etc
 
 my %typeSizes = (
        i8 => 'byte', u8 => 'byte',
@@ -29,6 +37,26 @@ my %typeSizes = (
        f64 => 'double',
        );
 
+# or just use some formatting function table
+my %defineType = (
+       %typeSizes,
+       string => 'String'
+       );
+
+my %definePrefix = (
+       i8 => '(byte)',
+       u8 => '(byte)',
+       i16 => '(short)',
+       u16 => '(short)',
+       string => '"'
+       );
+my %defineSuffix = (
+       u64 => 'L',
+       i64 => 'L',
+       f32 => 'f',
+       string => '"'
+       );
+
 my %intSizes = ( 8 => 'byte', 16 => 'short', 32 => 'int', 64 => 'long' );
 my %typePrimitive = (
        "byte" => 8,
@@ -50,28 +78,337 @@ my %typeSignature = (
        "MemoryAddress" => "Ljdk/incubator/foreign/MemoryAddress;",
        );
 
+my %renameTable = (
+       'studly-caps' => sub { my $s = shift; $s =~ s/(?:^|_)(.)/\U$1/g; return $s; },
+       'camel-case' => sub { my $s = shift; $s =~ s/(?:_)(.)/\U$1/g; return $s; },
+       'upper-leadin' => sub { my $s = shift; $s =~ s/^([^_]+)/\U$1/; return $s; },
+       'identity' => sub {     return $_[0]; },
+       );
+
+my %data = ();
+
 while (@ARGV) {
-       my $cmd = shift(@ARGV);
+       my $cmd = shift;
 
        if ($cmd eq "-t") {
-               $package = shift(@ARGV);
+               $package = shift;
     } elsif ($cmd eq "-d") {
-               $output = shift(@ARGV);
+               $output = shift;
     } elsif ($cmd eq "-v") {
                $verbose++;
+    } elsif ($cmd eq "-a") {
+               my $file = shift;
+               my $info;
+
+               print "Add $file\n";
+
+               unless ($info = do $file) {
+                       die "couldn't parse $file: $@" if $@;
+                       die "couldn't import $file: $!"    unless defined $info;
+                       die "couldn't run $file"       unless $info;
+               }
+
+               %data = (%data, %{$info});
        } else {
-               $meta = $cmd;
+               $apidef = $cmd;
        }
 }
 
-# load in interface file
-do $meta;
+my $api = loadControlFile($apidef);
 
+analyseAPI($api);
 analyseAndFixTypes();
 
-if ($verbose) {
+if (0) {
+       $s = $data{'struct:AVCodecContext'};
+       $obj = findAPIObject($api, 'struct', $s->{name});
+       print Dumper($obj);
+
+       foreach $m (@{$s->{fields}}) {
+               $inc = findAPIField($obj, $m->{name});
+               print " $m->{name} - $inc\n";
+       }
+}
+
+my $toDump = analyseDependencies(\%data, findRoots($api));
+
+if ($verbose > 1) {
        print "Using:n";
        print Dumper(\%data);
+       print "API:n";
+       print Dumper($api);
+}
+
+# find api.struct that matches a given struct name
+sub findAPIObject {
+       my $api = shift;
+       my $type = shift;
+       my $name = shift;
+
+       print "find api for $type:$name\n" if $verbose;
+       foreach $obj ( @{$api->{$type}} ) {
+               next if $obj->{name} eq '<default>';
+               print " $obj->{name} ? $name\n" if $verbose;
+               if ($obj->{name} =~ m@/(.*)/@) {
+                       my $rx = qr/$1/;
+                       return $obj if ($name =~ m/$rx/);
+               } elsif ($obj->{name} eq $name) {
+                       return $obj;
+               }
+       }
+
+       print " -> fallback=$type:<default>\n" if $verbose && $api->{"$type:<default>"};
+       return $api->{"$type:<default>"};
+}
+
+# sub findAPIStruct {
+#      my $api = shift;
+#      my $name = shift;
+
+#      foreach $obj ( @{$api->{struct}} ) {
+#              next if $obj->{name} eq '<default>';
+#              print "$obj->{name} ? $name\n" if $verbose;
+#              if ($obj->{name} =~ m@/(.*)/@) {
+#                      my $rx = qr/$1/;
+#                      return $obj if ($name =~ m/$rx/);
+#              } elsif ($obj->{name} eq $name) {
+#                      return $obj;
+#              }
+#      }
+
+#      return $api->{'struct:<default>'};
+# }
+
+sub findAPIField {
+       my $obj = shift;
+       my $name = shift;
+
+       foreach $inc (grep { $_->{mode} eq 'field' } @{$obj->{items}}) {
+               return $inc if $name =~ m/$inc->{regex}/;
+       }
+}
+
+# find all directly referenced types and field types from api
+sub findRoots {
+       my $api = shift;
+       my %seen;
+
+       foreach $obj ( @{$api->{library}} ) {
+               foreach $inc ( @{$obj->{items}} ) {
+                       if ($inc->{mode} eq 'func') {
+                               my @list = grep { $_->{type} eq $inc->{mode} && $_->{name} =~ m/$inc->{regex}/ } values %data;
+
+                               foreach $func (@list) {
+                                       $seen{"func:$func->{name}"} ++;
+                               }
+                       }
+               }
+       }
+
+       # all defines included
+       foreach $def ( @{$api->{define}} ) {
+               $seen{"define:$def->{name}"} ++;
+       }
+
+       foreach $obj ( @{$api->{struct}}, @{$api->{func}}) {
+               my @list;
+
+               if ($obj->{name} =~ m@/(.*)/@) {
+                       my $rx = "$obj->{type}:$1";
+                       push @list, grep { $_ =~ m/$rx/ } keys %data;
+               } else {
+                       push @list, "$obj->{type}:$obj->{name}";
+               }
+
+               foreach $n (@list) {
+                       $seen{$n} ++;
+               }
+
+       }
+
+       delete $seen{'struct:<default>'};
+       delete $seen{'func:<default>'};
+       delete $seen{'call:<default>'};
+
+       my @list = sort keys %seen;
+
+       return \@list;
+}
+
+# analyse dependencies of the supplied roots
+# only fields actually referenced in the api.def file are included
+# \%seen = \%data, \@roots
+sub analyseDependencies {
+       my $data = shift;
+       my @roots = @{shift @_};
+       my %seen;
+
+       print "Finding dependencies of $#roots roots\n";
+
+       while ($#roots >= 0) {
+               my $name = shift @roots;
+               my $s = $data{$name};
+               my @list;
+
+               next if $seen{$name}++;
+
+               print "visit $name $s->{name}\n" if $verbose;
+
+               if ($s->{type} =~ m/struct|union/) {
+                       my $obj = findAPIObject($api, 'struct', $s->{name});
+                       if ($obj->{default} eq 'all') {
+                               push @list, @{$s->{fields}};
+                       } else {
+                               push @list, grep { findAPIField($obj, $_->{name}) } @{$s->{fields}};
+                       }
+               } elsif ($s->{type} =~ m/func|call/) {
+                       @list = @{$s->{arguments}};
+                       push @list, $s->{result};
+               }
+
+               foreach $m (@list) {
+                       my $type = $m->{type};
+
+                       print " item $m->{name} '$type'\n" if $verbose;
+                       if ($m->{ctype} =~ m/enum (.*)/) {
+                               $type = "enum:$1";
+                       }
+
+                       push @roots,$type if $data{$type};
+               }
+       }
+
+       foreach $name (sort grep { m/:/ } keys %seen) {
+               print " $name\n";
+       }
+       print "\n";
+       return \%seen;
+}
+
+# find which api->thing->items applies to a given field name, if any
+sub findAPIItem {
+       my $api = shift;
+       my $type = shift;
+       my $target = shift;
+       my $mode = shift;
+       my $name = shift;
+
+       #print "search for $target.$name in $type.$mode\n";
+       # what about exclude?
+       foreach $obj ( @{$api->{$type}} ) {
+               if ($obj->{name} eq $target) {
+                       #print " found $target\n";
+                       foreach $inc (grep { $_->{mode} eq $mode } @{$obj->{items}}) {
+                               #print "  check $inc->{match}\n";
+                               return $inc if $name =~ m/$inc->{regex}/;
+                       }
+               }
+       }
+}
+
+sub analyseAPI {
+       my $api = shift;
+
+       foreach $obj ( @{$api->{struct}}, @{$api->{library}}, @{$api->{func}}) {
+               $obj->{access} = 'rw';
+               $obj->{default} = 'all';
+               $obj->{rename} = $renameTable{'identity'};
+               $obj->{'func:rename'} = $renameTable{'identity'};
+               $obj->{'field:rename'} = $renameTable{'identity'};
+               foreach $o (@{$obj->{options}}) {
+                       if ($o =~ m/^default=(none|all)$/) {
+                               $obj->{default} = $1;
+                       } elsif ($o =~ m/^access=([rwi]+)$/) {
+                               $obj->{access} = $1;
+                       } elsif ($o =~ m@^(rename|field:rename|func:rename)=(.*)@) {
+                               my $target = $1;
+
+                               if ($obj->{name} eq 'SwsContext') {
+                                       print "SwsContext rename = $o\n";
+                               }
+
+                               foreach $n (split /,/,$2) {
+                                       my $old = $obj->{$target};
+                                       my $new = $renameTable{$n};
+
+                                       if ($n =~ m@^s/(.*)/(.*)/$@) {
+                                               my $rx = qr/$1/;
+                                               my $rp = $2;
+                                               $obj->{$target} = sub { my $s=shift; $s = $old->($s); $s =~ s/$rx/$rp/; return $s;};
+                                       } elsif ($new) {
+                                               $obj->{$target} = sub { my $s=shift; $s = $old->($s); return $new->($s); };
+                                       }
+                               }
+                       }
+               }
+
+               my $defmode = $obj->{type} eq 'library' ? 'func' : 'field';
+
+               foreach $inc (@{$obj->{items}}) {
+                       if ($inc->{match} =~ m@^(field|func|define|struct|enum):/(.*)/$@) {
+                               $inc->{regex} = qr/$2/;
+                               $inc->{mode} = $1; # ?? "$1-include";
+                       } elsif ($inc->{match} =~ m@^(field|func|define|struct|enum):(.*)$@) {
+                               $inc->{regex} = qr/^$2$/;
+                               $inc->{mode} = $1;
+                       } elsif ($inc->{match} =~ m@^/(.*)/$@) {
+                               $inc->{regex} = qr/$1/;
+                               $inc->{mode} = $defmode;
+                       } else {
+                               $inc->{regex} = qr/^$inc->{match}$/;
+                               $inc->{mode} = $defmode;
+                       }
+
+                       $inc->{rename} = $renameTable{'identity'};
+                       $inc->{scope} = 'static' if $obj->{type} eq 'library';
+
+                       # maybe depends on mode above
+                       foreach $o (@{$inc->{options}}) {
+                               if ($o =~ m/^access=([rwi])+/) {
+                                       $inc->{access} = $1;
+                               } elsif ($o =~ m/^rename=(.*)/) {
+                                       foreach $n (split /,/,$1) {
+                                               my $old = $inc->{rename};
+                                               my $new = $renameTable{$n};
+
+                                               if ($n =~ m@^s/(.*)/(.*)/$@) {
+                                                       my $rx = qr/$1/;
+                                                       my $rp = $2;
+                                                       $inc->{rename} = sub { my $s=shift; $s = $old->($s); $s =~ s/$rx/$rp/; return $s;};
+                                               } elsif ($new) {
+                                                       $inc->{rename} = sub { my $s=shift; $s = $old->($s); return $new->($s); };
+                                               } else {
+                                                       my $x = $n;
+                                                       $inc->{rename} = sub { return $x; };
+                                               }
+                                       }
+                               } elsif ($o =~ m/^array-size=(.*)/) {
+                                       $inc->{'array_size'} = $1;
+                               } elsif ($o =~ m/^array$/) {
+                                       $inc->{'array'} = 1;
+                               } elsif ($o =~ m/^instance=(.*)/) {
+                                       $inc->{instance} = $1;
+                               } elsif ($o =~ m/^static$/) {
+                                       $inc->{scope} = 'static';
+                               } elsif ($o =~ m/^constructor=(.*)$/) {
+                                       $inc->{constructor} = $1;
+                               } elsif ($o =~ m/^constructor-result=(.*)$/) {
+                                       $inc->{constructor_result} = $1;
+                               } elsif ($o =~ m/^success=(.*)$/) {
+                                       $inc->{success} = $1;
+                               }
+                               # exclude mode, etc
+                       }
+
+                       $inc->{rename} = $obj->{"$inc->{mode}:rename"} if $inc->{rename} == $renameTable{'identity'} && $obj->{"$inc->{mode}:rename"};
+               }
+
+               if ($obj->{name} eq '<default>') {
+                       $api->{"$obj->{type}:<default>"} = $obj;
+               }
+       }
+
+       $api->{'call:<default>'} = { rename => $renameTable{'identity'}, scope => 'static'} if !$api->{'call:<default>'};
 }
 
 # anonymous structs
@@ -104,7 +441,6 @@ sub analyseAndFixTypes {
                my $s = $data{$n};
                my @list;
 
-
                if ($s->{type} =~ m/struct|union/) {
                        @list = @{$s->{fields}};
                } elsif ($s->{type} =~ m/func|call/) {
@@ -127,7 +463,7 @@ sub analyseAndFixTypes {
                        }
 
                        # must be last
-                       $a->{typeInfo} = queryTypeInfo($a);
+                       $a->{typeInfo} = analyseTypeInfo($s, $a);
                }
        }
 
@@ -158,14 +494,18 @@ sub formatTypeLayout {
        if ($m->{deref} =~ m/^(u64|u32):/) {
                $desc .= "Memory.POINTER$withName";
        } elsif ($m->{type} =~ m/^([iuf]\d+)$/) {
-               if ($m->{deref} =~ m/\[(\d*).*\]/) {
+               if ($m->{deref} =~ m/\[(\d*)u64:.*\]/) {
+                       $desc .= "MemoryLayout.sequenceLayout($1, Memory.POINTER)$withName";
+               } elsif ($m->{deref} =~ m/\[(\d*).*\]/) {
                        $desc .= "MemoryLayout.sequenceLayout($1, Memory.".uc($typeSizes{$m->{type}}).")$withName";
                } else {
                        $desc .= 'Memory.'.uc($typeSizes{$m->{type}})."$withName";
                }
        } elsif ($m->{type} =~ m/^(struct|union):(.*)/) {
                my $type = $2;
-               if ($m->{deref} =~ m/\[(\d*).*\]/) {
+               if ($m->{deref} =~ m/\[(\d*)u64:.*\]/) {
+                       $desc .= "MemoryLayout.sequenceLayout($1, Memory.POINTER)$withName";
+               } elsif ($m->{deref} =~ m/\[(\d*).*\]/) {
                        $desc .= "MemoryLayout.sequenceLayout($1, $type.LAYOUT)$withName";
                } else {
                        $desc .= "$type.LAYOUT$withName";
@@ -222,9 +562,19 @@ sub formatSignature {
        return $desc;
 }
 
-sub queryTypeInfo {
-       my $m = shift @_;
+# TODO: perhaps ByteArray should just be MemorySegment, kinda painful to wrap them all the time
+sub analyseTypeInfo {
+       my $s = shift;
+       my $m = shift;
        my $info = {};
+       my $inc;
+
+       #print " query $s->{name} $s->{type} '$m->{name}', '$m->{type}'\n";
+       if ($s->{type} eq 'struct') {
+               $inc = findAPIItem($api, 'struct', $s->{name}, 'field', $m->{name});
+       } elsif ($s->{type} eq 'func') {
+               $inc = findAPIItem($api, 'func', $s->{name}, 'field', $m->{name});
+       }
 
        # default for everything not specifically handled
        $info->{carrier} = "MemoryAddress";
@@ -234,21 +584,43 @@ sub queryTypeInfo {
                # This is a function pointer, type must be type = 'call:.*'
                if ($m->{type} =~ m/^call:(.*)/) {
                        $info->{type} = "Memory.FunctionPointer<$1>";
-                       $info->{create} = "$1.downcall(\${result}, scope())";
+                       $info->{create} = "$1.downcall(\${result}, \${scope})";
                } else {
                        die();
                }
        } elsif ($m->{type} =~ m/^([iuf]\d+)$/) {
-               if ($m->{deref} =~ m/\[(\d*).*\]/) {
+               if ($m->{deref} =~ m/\[(\d*)u64:.*\]/) {
+                       $info->{byValue} = 1;
+                       $info->{type} = "Memory.PointerArray";
+                       $info->{create} = $info->{type}.".create(\${result})";
+               } elsif ($m->{deref} =~ m/\[(\d*).*\]/) {
+                       # TODO: some mode thing rather than byvalue?
                        $info->{byValue} = 1;
                        $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
                        $info->{create} = $info->{type}.".create(\${result})";
                } elsif ($m->{deref} =~ m/^(u64:u64:|u32:u32:)/) {
                        $info->{type} = "Memory.PointerArray";
-                       $info->{create} = $info->{type}.".create(\${result})";
+                       $info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})";
                } elsif ($m->{deref} =~ m/^(u64:|u32:)/) {
-                       $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
-                       $info->{create} = $info->{type}.".create(\${result})";
+                       # assume any char * is a string unless an array-size or array is specified
+                       if ($inc->{array}) {
+                               $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
+                               $info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})";
+                       } elsif ($inc->{array_size}) {
+                               $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
+                               $info->{create} = $info->{type}.".createArray(\${result}, \${array_size}, \${scope})";
+                       } elsif ($typeSizes{$m->{type}} eq 'byte') {
+                               $info->{type} = 'String';
+                               $info->{resolve} = "(Addressable)Memory.address(frame.copy(\${value}))";
+                               $info->{create} = "(\${result}).getUtf8String(0)";
+                               $info->{resolveFrame} = 1;
+                               # for a function or a constructor that uses this element
+                               $s->{resolveFrame} = 1;
+                       } else {
+                               # ideally length 0 but panama-foreign doesn't grok that so fuckit
+                               $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
+                               $info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})";
+                       }
                } else {
                        $info->{type} = $typeSizes{$m->{type}};
                        $info->{carrier} = $typeSizes{$m->{type}};
@@ -257,23 +629,42 @@ sub queryTypeInfo {
                }
        } elsif ($m->{type} =~ m/^(struct|union):(.*)/) {
                my $type = $2;
-               if ($m->{deref} =~ m/\[(\d*).*\]/) {
-                       $info->{type} = $type;
-                       $info->{create} = $info->{type}.".createArray($1, \${result}, scope())";
+               if ($m->{deref} =~ m/\[(\d*)(.*)\]/) {
+                       $info->{byValue} = 1;
+                       # handle  'type name[x]' and  'type *name[x]'
+                       my $count = $1;
+                       my $deref = $2;
+                       if ($deref =~ m/^u64:u64:/) {
+                               die("can't handle double-deref array");
+                       } elsif ($deref =~ m/^u64:/) {
+                               $info->{type} = "Memory.HandleArray<$type>";
+                               $info->{create} = "Memory.HandleArray.create(\${result}, $type\:\:create, \${scope})";
+                       } else {
+                               $info->{type} = $type;
+                               #$info->{create} = $info->{type}.".create(\${result}, \${scope})";
+                               $info->{create} = $info->{type}.".create(\${result})";
+                       }
                } elsif ($m->{deref} =~ m/^(u64:u64:|u32:u32:)/) {
-                       $info->{type} = "Memory.PointerArray";
-                       $info->{create} = $info->{type}.".create(\${result})";
+                       # this assumes ** is as far as it gets
+                       $info->{type} = "Memory.HandleArray<$type>";
+                       if ($inc->{array_size}) {
+                               $info->{create} = "Memory.HandleArray.createArray(\${result}, \${array_size}, $type\:\:create, \${scope})";
+                       } else {
+                               $info->{create} = "Memory.HandleArray.createArray(\${result}, Long.MAX_VALUE, $type\:\:create, \${scope})";
+                       }
                } elsif ($m->{deref} =~ m/^(u64:|u32:)/) {
                        $info->{type} = $type;
-                       $info->{create} = $info->{type}.".create(\${result}, scope())";
+                       $info->{create} = $info->{type}.".create(\${result}, \${scope})";
                } else {
+                       # FIXME: change this to a reftype or something
+                       $info->{byValue} = 1;
                        $info->{type} = $type;
                        $info->{create} = $info->{type}.".create(\${result})";
                }
        } elsif ($m->{type} eq "void") {
                if ($m->{deref} =~ m/^(u64:u64:|u32:u32:)/) {
                        $info->{type} = "Memory.PointerArray";
-                       $info->{create} = $info->{type}.".create(\${result})";
+                       $info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})";
                } elsif ($m->{deref} =~ m/^(u64:|u32:)/) {
                        $info->{type} = "MemoryAddress";
                        $info->{create} = "\${result}";
@@ -291,45 +682,160 @@ sub queryTypeInfo {
        return $info;
 }
 
-sub formatFunction {
-       my $c = shift @_;
+# TODO: see if this can be merged with formatFunction
+# TODO: should constructor take a resourcescope always?
+# TODO: exceptions for errors
+sub formatConstructor {
+       my $c = shift;
+       my $inc = shift;
        my @arguments = @{$c->{arguments}};
        my $result = $c->{result};
        my $desc;
        my $index = 0;
-       my $desc;
+       my $count = 0;
+       my $desc = "";
        my $name = $c->{name};
-
        my $rtype = $result->{typeInfo}->{type};
+       my $otype = $data{$arguments[$inc->{constructor_result}]->{type}}->{name};
 
-       #print Dumper($c);
 
-       $desc = $rtype;
+       $name = $inc->{rename}->($name);
+
+       $desc .= "static " if $inc->{scope} eq 'static';
+       $desc .= $otype;
        $desc .= " $name(";
 
        for $m (@arguments) {
-               $desc .= ", " if ($index++ > 0);
-               $desc .= $m->{typeInfo}->{type};
-               $desc .= " $m->{name}"
+               if (($inc->{constructor} && $index != $inc->{constructor_result})) {
+                       $desc .= ", " if ($count++ > 0);
+                       $desc .= $m->{typeInfo}->{type};
+                       $desc .= " $m->{name}"
+               }
+               $index++;
+       }
+       if ($inc->{constructor}) {
+               $desc .= ", " if ($count++ > 0);
+               $desc .= "ResourceScope scope";
        }
        $desc .=") {\n ";
+
+       $desc .= "$result->{typeInfo}->{carrier} res\$value;\n" if ($rtype ne "void");
+
+       $desc .= " try ";
+       $desc .= "(Frame frame = Memory.createFrame()) " if ($c->{resolveFrame});
+       $desc .= "{\n";
+       $desc .= "   Memory.HandleArray<$otype> res\$holder = Memory.HandleArray.createArray(1, frame, $otype\:\:create, scope);\n" if ($inc->{constructor});
+       $desc .= "   res\$value = ($result->{typeInfo}->{carrier})" if ($rtype ne "void");
+       $desc .= "   " if ($rtype eq "void");
+
        $index = 0;
+       $desc .= "$c->{name}\$FH.invokeExact(\n   ";
+       for $m (@arguments) {
+               my $resolve = $m->{typeInfo}->{resolve};
+
+               if ($inc->{constructor} && $index == $inc->{instance}) {
+                       $desc .= ",\n    " if ($index++ > 0);
+                       $desc .= "(Addressable)res\$holder.address()";
+               } elsif ($inc->{scope} ne 'static' && $index == $inc->{instance}) {
+                       $desc .= ",\n    " if ($index++ > 0);
+                       $desc .= "(Addressable)address()";
+               } else {
+                       $desc .= ",\n    " if ($index++ > 0);
+
+                       if ($resolve) {
+                               $resolve =~ s/\$\{value\}/$m->{name}/g;
+                               $desc .= $resolve;
+                       } else {
+                               $desc .= "$m->{name}";
+                       }
+               }
+       }
+       $desc .= ");\n";
+
+       if ($rtype ne "void" && defined $inc->{success}) {
+       #       my $create = $result->{typeInfo}->{create};
+
+       #       # ooh, templates could insert other arguments or values as well?
+       #       $create =~ s/\$\{result\}/res\$value/;
+       #       if ($inc->{scope} eq 'static') {
+       #               $create =~ s/\$\{scope\}/ResourceScope.globalScope()/;
+       #       } else {
+       #               $create =~ s/\$\{scope\}/scope()/;
+       #       }
+
+               foreach $code (split /,/,$inc->{success}) {
+                       $desc .= "   if (res\$value == $code) return res\$holder.getAtIndex(0);\n";
+               }
+       } else {
+               $desc .= "   return res\$holder.getAtIndex(0);\n";
+       }
+       # throw Error()?
+       $desc .= " } catch (Throwable t) { throw new RuntimeException(t); }\n";
+
+       # throw failures here based on res$value
+       $desc .= " return null;\n";
+
+       $desc .="}";
+
+       #print "$desc\n";
+       return $desc;
+}
+
+sub formatFunction {
+       my $c = shift;
+       my $inc = shift;
+       my @arguments = @{$c->{arguments}};
+       my $result = $c->{result};
+       my $desc;
+       my $index = 0;
+       my $count = 0;
+       my $desc = "";
+       my $name = $c->{name};
+       my $rtype = $result->{typeInfo}->{type};
+
+       if ($inc->{constructor}) {
+               return formatConstructor($c, $inc);
+       }
+
+       $name = $inc->{rename}->($name);
+
+       $desc .= "static " if $inc->{scope} eq 'static';
+       $desc .= $rtype;
+       $desc .= " $name(";
+
+       for $m (@arguments) {
+               if ($inc->{scope} eq 'static' || $index != $inc->{instance}) {
+                       $desc .= ", " if ($count++ > 0);
+                       $desc .= $m->{typeInfo}->{type};
+                       $desc .= " $m->{name}"
+               }
+               $index++;
+       }
+       $desc .=") {\n ";
 
-       $desc .= "try {\n";
+       $desc .= "try ";
+       $desc .= "(Frame frame = Memory.createFrame()) " if ($c->{resolveFrame});
+       $desc .= "{\n";
        $desc .= "  $result->{typeInfo}->{carrier} res\$value = ($result->{typeInfo}->{carrier})" if ($rtype ne "void");
        $desc .= "  " if ($rtype eq "void");
 
-       $desc .= "$name\$FH.invokeExact(\n   ";
+       $index = 0;
+       $desc .= "$c->{name}\$FH.invokeExact(\n   ";
        for $m (@arguments) {
                my $resolve = $m->{typeInfo}->{resolve};
 
-               $desc .= ",\n   " if ($index++ > 0);
-
-               if ($resolve) {
-                       $resolve =~ s/\$\{value\}/$m->{name}/g;
-                       $desc .= $resolve;
+               if ($inc->{scope} ne 'static' && $index == $inc->{instance}) {
+                       $desc .= ",\n   " if ($index++ > 0);
+                       $desc .= "(Addressable)address()";
                } else {
-                       $desc .= "$m->{name}"
+                       $desc .= ",\n   " if ($index++ > 0);
+
+                       if ($resolve) {
+                               $resolve =~ s/\$\{value\}/$m->{name}/g;
+                               $desc .= $resolve;
+                       } else {
+                               $desc .= "$m->{name}";
+                       }
                }
        }
        $desc .= ");\n";
@@ -338,7 +844,12 @@ sub formatFunction {
                my $create = $result->{typeInfo}->{create};
 
                # ooh, templates could insert other arguments or values as well?
-               $create =~ s/\${result\}/res\$value/;
+               $create =~ s/\$\{result\}/res\$value/;
+               if ($inc->{scope} eq 'static') {
+                       $create =~ s/\$\{scope\}/ResourceScope.globalScope()/;
+               } else {
+                       $create =~ s/\$\{scope\}/scope()/;
+               }
 
                $desc .= "  return $create;\n";
        }
@@ -351,9 +862,10 @@ sub formatFunction {
 }
 
 # create an interface for function pointers
-# FiXME: this should be exportCallback to a file
+# FiXME: this should be exportCallback to a file?
 sub formatCallback {
-       my $c = shift @_;
+       my $c = shift;
+       my $obj = shift;
        my @arguments = @{$c->{arguments}};
        my $result = $c->{result};
        my $desc;
@@ -421,6 +933,7 @@ sub formatCallback {
                my $create = $m->{typeInfo}->{create};
 
                $create =~ s/\$\{result\}/$m->{name}/g;
+               $create =~ s/\$\{scope\}/scope/g;
 
                $desc .= ",\n    " if ($index++ > 0);
                $desc .= "$create";
@@ -450,12 +963,13 @@ sub formatCallback {
        $desc .= "   symbol,\n";
 
        # HACK: this is basically the same as any function call, just patch in the changes for now
-       $tmp = formatFunction($c);
+       $tmp = formatFunction($c, $obj);
 
        $tmp =~ s/^(.*) ($name)\(/(/;
        $tmp =~ s/\) \{/) -> {/;
        $tmp =~ s/^/   /mg;
        $desc .= $tmp;
+
        $desc .= "\n";
        $desc .= "  );\n";
        $desc .= " }\n";
@@ -591,72 +1105,91 @@ sub formatLayout {
 }
 
 sub formatGetSet {
-       my $s = shift @_;
-       my $m = shift @_;
+       my $s = shift;
+       my $m = shift;
+       my $rename = shift;
+       my $access = shift;
+       my $inc = shift;
        my $desc = "";
        my $info = $m->{typeInfo};
-       my $Name = ucfirst($m->{name});
+       my $Name = ucfirst($rename);
        my $tmp;
 
        # info -> needsalloc?
 
-       # TODO: String
        # TODO: embedded arrays are quite different setup
 
        if ($info->{byValue}) {
                $tmp = $info->{create};
                $tmp =~ s/\$\{result\}/segment/g;
+               $tmp =~ s/\$\{scope\}/scope()/g;
 
                $desc .= " public $info->{type} get$Name() {\n";
-               $desc .= "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
-               $desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
+               #$desc .= "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
+               #$desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
+               $desc .= "  MemorySegment segment = this.segment.asSlice($m->{name}\$byteOffset, $m->{name}\$byteSize);\n";
                $desc .= "  return $tmp;\n";
                $desc .= " }\n";
 
-               $desc .= " public $info->{type} get$Name"."At(long index) {\n";
-               $desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
-               $desc .= "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
-               $desc .= "  segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
-               $desc .= "  return $tmp;\n";
-               $desc .= " }\n";
-       } else {
+               if ($access =~ m/i/) {
+                       $desc .= " public $info->{type} get$Name"."At(long index) {\n";
+                       #$desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
+                       #$desc .= "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
+                       #$desc .= "  segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
+                       $desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index + $m->{name}\$byteOffset, $m->{name}\$byteSize);\n";
+                       $desc .= "  return $tmp;\n";
+                       $desc .= " }\n";
+               }
+       } elsif ($access =~ /r/) {
                $tmp = $info->{create};
+
                $tmp =~ s/\$\{result\}/($info->{carrier})$m->{name}\$VH.get(segment)/g;
+               $tmp =~ s/\$\{scope\}/scope()/g;
+               # fixme: lookup type of array size? somewhere?  doesn't matter i think
+               $tmp =~ s/\${array_size}/(long)$inc->{array_size}\$VH.get(segment)/g;
 
                $desc .= " public $info->{type} get$Name() {\n";
                $desc .= "  return $tmp;\n";
                $desc .= " }\n";
 
-               $desc .= " public $info->{type} get$Name"."At(long index) {\n";
-               $desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
-               $desc .= "  return $tmp;\n";
-               $desc .= " }\n";
+               if ($access =~ m/i/) {
+                       $desc .= " public $info->{type} get$Name"."At(long index) {\n";
+                       $desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
+                       $desc .= "  return $tmp;\n";
+                       $desc .= " }\n";
+               }
        }
 
-       if (!($m->{deref} =~ m/\[(\d*).*\]/)) {
+       if ($access =~ m/w/ && !$info->{byValue}) {
                $tmp = $info->{resolve};
                $tmp =~ s/\$\{value\}/value/g;
 
                $desc .=  " public void set$Name($info->{type} value) {\n";
+               $desc .=  " try (Frame frame = Memory.createFrame()) {\n" if ($info->{resolveFrame});
                $desc .=  "  $m->{name}\$VH.set(segment, $tmp);\n";
+               $desc .=  " }\n" if ($info->{resolveFrame});
                $desc .=  " }\n";
 
-               $desc .=  " public void set$Name"."At(long index, $info->{type} value) {\n";
-               $desc .=  "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
-               $desc .=  "  $m->{name}\$VH.set(segment, $tmp);\n";
-               $desc .=  " }\n";
+               if ($access =~ m/i/) {
+                       $desc .=  " public void set$Name"."At(long index, $info->{type} value) {\n";
+                       $desc .=  " try (Frame frame = Memory.createFrame()) {\n" if ($info->{resolveFrame});
+                       $desc .=  "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
+                       $desc .=  "  $m->{name}\$VH.set(segment, $tmp);\n";
+                       $desc .=  " }\n" if ($info->{resolveFrame});
+                       $desc .=  " }\n";
+               }
        }
 
-       # indexed
-
        return $desc;
 }
 
 sub exportStruct {
-       my $f = shift @_;
-       my $s = shift @_;
+       my $f = shift;
+       my $s = shift;
+       my $obj = shift;
        my @fields = @{$s->{fields}};
        my $isHandle = $s->{size} == 0;
+       my $doArray = $obj->{access} =~ m/i/;
        #my @functions = @{shift @_};
 
        print $f "package $package;\n" if $package;
@@ -673,11 +1206,13 @@ sub exportStruct {
                print $f " private $s->{name}(MemorySegment segment) { this.segment = segment; }\n";
                print $f " public static $s->{name} create(MemorySegment segment) { return new $s->{name}(segment); }\n";
                print $f " public static $s->{name} create(MemoryAddress address, ResourceScope scope) {\n";
-               print $f "  return create(MemorySegment.ofAddress(address, LAYOUT.byteSize(), scope));\n";
-               print $f " }\n";
-               print $f " public static $s->{name} createArray(MemoryAddress address, long size, ResourceScope scope) {\n";
-               print $f "  return create(MemorySegment.ofAddress(address, size * LAYOUT.byteSize(), scope));\n";
+               print $f "  return MemoryAddress.NULL != address ? create(MemorySegment.ofAddress(address, LAYOUT.byteSize(), scope)) : null;\n";
                print $f " }\n";
+               if ($doArray) {
+                       print $f " public static $s->{name} createArray(MemoryAddress address, long size, ResourceScope scope) {\n";
+                       print $f "  return MemoryAddress.NULL != address ? create(MemorySegment.ofAddress(address, size * LAYOUT.byteSize(), scope)) : null;\n";
+                       print $f " }\n";
+               }
                print $f " public static $s->{name} create(Frame frame) { return create(frame.allocate(LAYOUT)); }\n";
                print $f " public static $s->{name} create(ResourceScope scope) { return create(MemorySegment.allocateNative(LAYOUT, scope)); }\n";
                print $f " public MemoryAddress address() { return segment.address(); }\n";
@@ -687,91 +1222,81 @@ sub exportStruct {
                print $f " MemoryAddress address;\n";
                print $f " ResourceScope scope;\n";
                # constructors
-               print $f " private $s->{name}(MemoryAddress address, ResourceScope scope) { this.address = address; this.scope = scope}\n";
-               print $f " public static $s->{name} create(MemoryAddress address) { return new $s->{name}(address); }\n";
+               print $f " private $s->{name}(MemoryAddress address, ResourceScope scope) { this.address = address; this.scope = scope;}\n";
+               print $f " public static $s->{name} create(MemoryAddress address, ResourceScope scope) { return MemoryAddress.NULL != address ? new $s->{name}(address, scope) : null; }\n";
                print $f " public MemoryAddress address() { return address; }\n";
                print $f " public ResourceScope scope() { return scope; }\n";
        }
 
-       # FIXME: use typeInfo
-       # TODO: indexed accessors
-       # accessors
-       if (1) {
-               foreach $m (@fields) {
-                       print $f formatGetSet($s, $m);
-               }
-       } else {
-               foreach $m (@fields) {
-                       my $Name = ucfirst($m->{name});
+       my %seen;
 
-                       print $f " // [$m->{deref}] [$m->{type}] [$m->{ctype}]\n";
+       # Any defines
+       foreach $inc (grep { $_->{mode} eq 'define' } @{$obj->{items}}) {
+               my $def = $data{$inc->{match}};
 
-                       if ($m->{deref} =~ m/^(u64:|u32:)\(/) {
-                               # This is a function pointer, type must be type = 'call:.*'
+               die ("unknown define $inc->{match} in $s->{name}\n") if !$def;
 
-                               if ($m->{type} =~ m/^call:(.*)/) {
-                                       my $jtype = $1;
+               delete $toDump->{$inc->{match}};
 
-                                       $jtype =~ s/(.*)\((.*)\)(.*)/Call$1_$2_$3/;
+               foreach $m (@{$def->{values}}) {
+                       print $f " /**\n ($m->{comment}) */\n" if ($m->{comment});
+                       print $f " public static final $defineType{$m->{type}} $m->{name} = $definePrefix{$m->{type}}$m->{value}$defineSuffix{$m->{type}};\n";
+               }
+       }
 
-                                       print $f " public Memory.FunctionPointer<$jtype> get$Name() {\n";
-                                       print $f "  // FIXME: better scope\n";
-                                       print $f "  return $jtype.downcall((MemoryAddress)$m->{name}\$VH.get(segment), scope());\n";
-                                       print $f " }\n";
+       # TODO: any enums we want to include here I suppose?
 
-                                       print $f " public void set$Name(Memory.FunctionPointer<$jtype> value) {\n";
-                                       print $f "  $m->{name}\$VH.set(segment, Memory.address(value));\n";
-                                       print $f " }\n";
-                               }
-                       } elsif ($m->{deref} =~ m/^(u64:|u32:)/) {
-                               # all other pointer types require extra context, e.g. a length or type
-                               print $f " public MemoryAddress get$Name() {\n";
-                               print $f "  return (MemoryAddress)$m->{name}\$VH.get(segment);\n";
-                               print $f " }\n";
-
-                               # FIXME: set could use the type though
-                               print $f " public void set$Name(MemoryAddress value) {\n";
-                               print $f "  $m->{name}\$VH.set(segment, Memory.address(value));\n";
-                               print $f " }\n";
-                       } elsif ($m->{type} eq "bitfield") {
-                               # TODO
-                       } elsif ($m->{type} =~ m/^struct|union:(.*)$/) {
-                               my $jtype = $1;
-
-                               $jtype = "Memory.HandleArray<$jtype>" if ($data{$m->{type}}->{size} == 0);
-
-                               # embedded type including arrays
-                               print $f " public $jtype get$Name() {\n";
-                               print $f "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
-                               print $f "  MemorySegment seg = segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
-                               print $f "  return $jtype.create(seg, $jtype::new);\n" if ($data{$m->{type}}->{size} == 0);
-                               print $f "  return $jtype.create(seg);\n" if ($data{$m->{type}}->{size} != 0);
-                               print $f " }\n";
-                       } elsif ($m->{type} =~ m/^[uif]\d+$/) {
-                               my $jtype = $typeSizes{$m->{type}};
-                               my $Jtype = ucfirst($jtype);
-                               my $JTYPE = uc($jtype);
-
-                               if ($m->{deref} =~ m/\[(\d*).*\]/) {
-                                       # array type
-                                       print $f " public Memory.$Jtype"."Array get$Name() {\n";
-                                       print $f "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
-                                       print $f "  MemorySegment seg = segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
-                                       print $f "  return new Memory.$Jtype"."Array(seg);\n";
-                                       print $f " }\n";
-                               } else {
-                                       # primitive type
-                                       print $f " public $jtype get$Name() {\n";
-                                       print $f "  return ($jtype)$m->{name}\$VH.get(segment);\n";
-                                       print $f " }\n";
-
-                                       print $f " public void set$Name($jtype value) {\n";
-                                       print $f "  $m->{name}\$VH.set(segment, ($jtype)value);\n";
-                                       print $f " }\n";
-                               }
+       # Accessors
+       foreach $m (@fields) {
+               my $access = $obj->{access};
+               my $rename = $obj->{'field:rename'};
+               my $matches = 0;
+               my $matchinc;
+
+               # check for match
+               foreach $inc (grep { $_->{mode} eq 'field' } @{$obj->{items}}) {
+                       $matches = $m->{name} =~ m/$inc->{regex}/;
+
+                       if ($matches) {
+                               $access = $inc->{access} if $inc->{access};
+                               $rename = $inc->{rename} if $inc->{rename} != $renameTable{'identity'};
+                               $matchinc = $inc;
+                               last;
                        }
-                       # struct                                print $f " MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
+               }
+
+               my $output = $matches || ($obj->{default} eq 'all');
+
+               if ($output) {
+                       my $name = $rename ? $rename->($m->{name}) : $m->{name};
 
+                       print $f formatGetSet($s, $m, $name, $access, $matchinc);
+               }
+       }
+
+       # Functions
+       foreach $inc (grep { $_->{mode} eq 'func' } @{$obj->{items}}) {
+               my @list;
+
+               print "$obj->{name} match $inc->{match} regex $inc->{regex}\n" if $verbose;
+
+               if ($data{$inc->{match}}) {
+                       push @list, $data{$inc->{match}};
+               } else {
+                       @list = grep { $_->{name} =~ m/$inc->{regex}/ } values %data;
+               }
+
+               foreach $c (@list) {
+                       my $tmp;
+
+                       next if $seen{$c->{name}}++;
+
+                       print $f " static final MethodHandle $c->{name}\$FH = Memory.downcall(\"$c->{name}\",\n";
+                       $tmp = formatFunctionDescriptor($c);
+                       print $f "$tmp);\n";
+
+                       $tmp = formatFunction($c, $inc);
+                       print $f 'public '.$tmp."\n\n";
                }
        }
 
@@ -780,11 +1305,59 @@ sub exportStruct {
                print $f "static final GroupLayout LAYOUT = ".formatLayout($s).";\n";
 
                foreach $m (@fields) {
-                       next if ($m->{typeInfo}->{byValue});
-                       print $f " static final VarHandle $m->{name}\$VH = LAYOUT.varHandle(MemoryLayout.PathElement.groupElement(\"$m->{name}\"));\n";
+                       print $f " // type='$m->{type}' deref='$m->{deref}'  info->type ='$m->{typeInfo}->{type}'\n";
+                       if ($m->{typeInfo}->{byValue}) {
+                               print $f " static final long $m->{name}\$byteOffset = "
+                                       ." LAYOUT.byteOffset(MemoryLayout.PathElement.groupElement(\"$m->{name}\"));\n";
+                               print $f " static final long $m->{name}\$byteSize = "
+                                       ."LAYOUT.select(MemoryLayout.PathElement.groupElement(\"$m->{name}\")).byteSize();\n";
+                       } else {
+                               print $f " static final VarHandle $m->{name}\$VH = "
+                                       ."LAYOUT.varHandle(MemoryLayout.PathElement.groupElement(\"$m->{name}\"));\n";
+                       }
                }
        }
 
+       # verification?
+       if (!$isHandle) {
+               print $f <<END;
+               static void check\$offset(String name, long bitoffset) {
+                       long byteoffset = bitoffset/8;
+                       long offset = LAYOUT.byteOffset(MemoryLayout.PathElement.groupElement(name));
+                       if (offset != byteoffset)
+                               throw new AssertionError(String.format("%s.offset %d != %d", name, byteoffset, offset), null);
+               }
+               {
+END
+               foreach $m (@fields) {
+                       print $f "check\$offset(\"$m->{name}\", $m->{offset});\n";
+               }
+my $bytes = $s->{size}/8;
+       print $f <<END;
+               if (LAYOUT.byteSize() != $bytes)
+                       throw new AssertionError(String.format("$s->{name}.sizeof = %d != $bytes", LAYOUT.byteSize()), null);
+       }
+END
+       }
+       print $f "}\n";
+}
+
+# file,enum
+sub exportEnum {
+       my $f = shift;
+       my $s = shift;
+       my @values = @{$s->{values}};
+       my $jtype = $typeSizes{$s->{value_type}};
+       my $prefix = $definePrefix{$s->{value_type}};
+       my $suffix = $definePrefix{$s->{value_type}};
+       print $f "package $package;\n" if $package;
+
+       print $f "public interface $s->{name} {\n";
+
+       foreach $v (@values) {
+               print $f " public static final $jtype $v->{name} = $prefix$v->{value}$suffix;\n";
+       }
+
        print $f "}\n";
 }
 
@@ -825,22 +1398,46 @@ sub nameToPath {
        return $name;
 }
 
-foreach $x (grep { m/^(struct|union):/ } sort keys %data) {
-       my $s = $data{$x};
-       my $path = nameToPath($output, "$package.$s->{name}");
+#print "api\n";
+#print Dumper($api);
 
-       open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
+# Dump struct type
+foreach $obj ( @{$api->{struct}} ) {
+       my @list;
 
-       exportStruct($f, $s);
+       next if $obj->{name} eq '<default>';
 
-       close $f;
-}
+       if ($obj->{name} =~ m@/(.*)/@) {
+               my $rx = qr/struct:$1/;
 
-foreach $x (grep { m/^call:/ } sort keys %data) {
-       my $c = $data{$x};
-       my $name = $c->{name};
+               @list = map { s/struct://; $_ } grep { $_ =~ m/$rx/ } keys %data;
+       } else {
+               push @list, $obj->{name};
+       }
 
-       my $path = nameToPath($output, "$package.$name");
+       foreach $name (@list) {
+               my $path = nameToPath($output, "$package.$name");
+               my $s = $data{"struct:$name"};
+
+               #print Dumper($obj);
+
+               delete $toDump->{"struct:$name"};
+
+               if ($s) {
+                       open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
+
+                       exportStruct($f, $s, $obj);
+
+                       close $f;
+               } else {
+                       print "No struct $name\n";
+               }
+       }
+}
+
+# Dump library type
+foreach $lib ( @{$api->{library}} ) {
+       my $path = nameToPath($output, "$package.$lib->{name}");
 
        open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
 
@@ -848,22 +1445,57 @@ foreach $x (grep { m/^call:/ } sort keys %data) {
        print $f "import jdk.incubator.foreign.*;\n";
        print $f "import java.lang.invoke.*;\n";
 
-       print $f formatCallback($c);
+       print $f "public class $lib->{name} {\n";
+
+       print $f " static ResourceScope scope() { return ResourceScope.globalScope(); }\n";
+
+       # scan for matches
+       foreach $inc (@{$lib->{items}}) {
+               if ($inc->{mode} eq 'func') {
+                       my @list = grep { $_->{type} eq $inc->{mode} && $_->{name} =~ m/$inc->{regex}/ } values %data;
+                       foreach $c (@list) {
+                               my $tmp;
+
+                               print $f " static final MethodHandle $c->{name}\$FH = Memory.downcall(\"$c->{name}\",\n";
+                               $tmp = formatFunctionDescriptor($c);
+                               print $f $tmp.");\n";
+
+                               $tmp = formatFunction($c, $inc);
+                               print $f "public ";
+                               print $f $tmp."\n\n";
+                       }
+               } elsif ($inc->{mode} eq 'define') {
+                       my @list = grep { $_->{type} eq $inc->{mode} && $_->{name} =~ m/$inc->{regex}/ } values %data;
+                       foreach $c (@list) {
+                               delete $toDump->{"define:$c->{name}"};
+                               foreach $m (@{$c->{fields}}) {
+                                       print $f " /**\n ($m->{comment}) */\n" if ($m->{comment});
+                                       print $f " public static final $defineType{$m->{type}} $m->{name} = $definePrefix{$m->{type}}$m->{value}$defineSuffix{$m->{type}};\n";
+                               }
+                       }
+               }
+       }
+
+       print $f "}\n";
 
        close $f;
 }
 
-# just quick and dirty for now
-# may want a non-static version with a specific scope?
-{
-       my @functions = grep { /^func:/ } keys %data;
-       my $lib = {
-               name => "APILib",
-               functions => \@functions
-       };
+print "remaining dependent types\n";
+foreach $k (sort grep { !m/func:/ } keys %{$toDump}) {
+       print " $k\n";
+}
+print "\n";
 
+# Calls referenced
+foreach $k (sort keys %{$toDump}) {
+       next if (!($k =~ m/call:(.+)/));
 
-       my $path = nameToPath($output, "$package.$lib->{name}");
+       my $name = $1;
+       my $c = $data{$k};
+       my $obj = findAPIObject($api, 'call', $name);
+
+       my $path = nameToPath($output, "$package.$name");
 
        open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
 
@@ -871,23 +1503,117 @@ foreach $x (grep { m/^call:/ } sort keys %data) {
        print $f "import jdk.incubator.foreign.*;\n";
        print $f "import java.lang.invoke.*;\n";
 
-       print $f "public class $lib->{name} {\n";
+       print $f formatCallback($c, $obj);
 
-       print $f " static ResourceScope scope() { return ResourceScope.globalScope(); }\n";
-       foreach $cname (@{$lib->{functions}}) {
-               my $c = $data{$cname};
-               my $tmp;
+       close $f;
+}
+
+# any struct remaining in toDump (but not in api)
+# FIXME: how to lookup obj?
+foreach $k (sort keys %{$toDump}) {
+       if ($k =~ m/struct:(.*)/) {
+               my $name = $1;
+               my $s = $data{$k};
+               my $path = nameToPath($output, "$package.$name");
+
+               die("missing struct $name") if !$s;
 
-               print $f " static final MethodHandle $c->{name}\$FH = Memory.downcall(\"$c->{name}\",\n";
-               $tmp = formatFunctionDescriptor($c);
-               print $f "$tmp);\n";
+               delete $toDump->{$k};
 
-               $tmp = formatFunction($c);
-               print $f "public static ";
-               print $f $tmp;
+               my $obj = findAPIObject($api, 'struct', $name);
+
+               open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
+               exportStruct($f, $s, $obj);
+               close $f;
        }
+}
 
-       print $f "}\n";
+# Dump enum types used by everything and not dumped elsehwere
+foreach $k (sort keys %{$toDump}) {
+       if ($k =~ m/enum:(.*)/) {
+               my $name = $1;
+               my $s = $data{$k};
+               my $path = nameToPath($output, "$package.$name");
 
-       close $f;
+               die("missing enum $name") if !$s;
+
+               open(my $f, ">", $path) || die ("Cannot open '$path' for writing");
+
+               exportEnum($f, $s);
+
+               close $f;
+       }
+}
+
+# Dump define types not dumped elsehwere
+foreach $k (sort keys %{$toDump}) {
+       if ($k =~ m/define:(.*)/) {
+               my $name = $1;
+               my $s = $data{$k};
+               my $path = nameToPath($output, "$package.$name");
+
+               die("missing define $name") if !$s;
+
+               open(my $f, ">", $path) || die ("Cannot open '$path' for writing");
+
+               print $f "package $package;\n" if $package;
+               print $f "public interface $s->{name} {\n";
+
+               foreach $m (@{$s->{values}}) {
+                       # some parsing problem here
+                       next if !$m->{value};
+
+                       print $f " /**\n ($m->{comment}) */\n" if ($m->{comment});
+                       print $f " public static final $defineType{$m->{type}} $m->{name} = $definePrefix{$m->{type}}$m->{value}$defineSuffix{$m->{type}};\n";
+               }
+
+               print $f "}\n";
+
+               close $f;
+       }
+}
+
+# and we're done
+exit 0;
+
+sub loadControlFile {
+       my $path = shift @_;
+       my %def = ();
+       my $target;
+
+       open (my $d,"<",$path);
+
+       while (<$d>) {
+               next if /\s*\#/;
+
+               chop;
+
+               if ($target) {
+                       if (m/\s*\}\s*$/) {
+                               undef $target;
+                       } elsif (/^\s*(\S+)\s*(.*)/) {
+                               my @options = split(/\s+/,$2);
+                               push @{$target->{items}}, {
+                                       match => $1,
+                                       options => \@options
+                               };
+                       }
+               } elsif (/^(\w+)\s+(\S*)\s*(.*)\s+\{/) {
+                       my @options = split(/\s+/,$3);
+
+                       $target = {
+                               type => $1,
+                               name => $2,
+                               options => \@options,
+                               items => []
+                       };
+                       push @{$def{$1}}, $target;
+               } elsif (/\S/) {
+                       die("invalid line: %_");
+               }
+       }
+
+       close $d;
+
+       return \%def;
 }
index 1cdf9f6..023c828 100644 (file)
@@ -181,7 +181,7 @@ public class Memory {
                        this.segment = segment;
                }
 
-               public static ByteArray create(MemorySegment segment) {
+               public static ByteArray create(MemorySegment segment) {
                        return new ByteArray(segment);
                }
 
@@ -242,16 +242,24 @@ public class Memory {
        public static class ShortArray extends AbstractList<Short> implements Memory.Addressable {
                final MemorySegment segment;
 
-               public ShortArray(MemorySegment segment) {
+               private ShortArray(MemorySegment segment) {
                        this.segment = segment;
                }
 
-               public ShortArray(Frame frame, long size) {
-                       this(frame.allocateArray(Memory.SHORT, size));
+               public static ShortArray create(MemorySegment segment) {
+                       return new ShortArray(segment);
                }
 
-               public ShortArray(Frame frame, short... values) {
-                       this(frame.allocateArray(Memory.SHORT, values));
+               public static ShortArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+                       return create(MemorySegment.ofAddress(address, length, scope));
+               }
+
+               public static ShortArray createArray(long length, SegmentAllocator alloc) {
+                       return create(alloc.allocateArray(Memory.SHORT, length));
+               }
+
+               public static ShortArray create(SegmentAllocator alloc, short... values) {
+                       return create(alloc.allocateArray(Memory.SHORT, values));
                }
 
                public final MemoryAddress address() {
@@ -295,16 +303,24 @@ public class Memory {
        public static class IntArray extends AbstractList<Integer> implements Memory.Addressable {
                final MemorySegment segment;
 
-               public IntArray(MemorySegment segment) {
+               private IntArray(MemorySegment segment) {
                        this.segment = segment;
                }
 
-               public IntArray(Frame frame, long size) {
-                       this(frame.allocateArray(Memory.INT, size));
+               public static IntArray create(MemorySegment segment) {
+                       return new IntArray(segment);
+               }
+
+               public static IntArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+                       return create(MemorySegment.ofAddress(address, length, scope));
+               }
+
+               public static IntArray createArray(long length, SegmentAllocator alloc) {
+                       return create(alloc.allocateArray(Memory.INT, length));
                }
 
-               public IntArray(Frame frame, int... values) {
-                       this(frame.allocateArray(Memory.INT, values));
+               public static IntArray create(SegmentAllocator alloc, int... values) {
+                       return create(alloc.allocateArray(Memory.INT, values));
                }
 
                public final MemoryAddress address() {
@@ -352,12 +368,20 @@ public class Memory {
                        this.segment = segment;
                }
 
-               public LongArray(Frame frame, long size) {
-                       this(frame.allocateArray(Memory.LONG, size));
+               public static LongArray create(MemorySegment segment) {
+                       return new LongArray(segment);
                }
 
-               public LongArray(Frame frame, long... values) {
-                       this(frame.allocateArray(Memory.LONG, values));
+               public static LongArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+                       return create(MemorySegment.ofAddress(address, length, scope));
+               }
+
+               public static LongArray createArray(long length, SegmentAllocator alloc) {
+                       return create(alloc.allocateArray(Memory.LONG, length));
+               }
+
+               public static LongArray create(SegmentAllocator alloc, long... values) {
+                       return create(alloc.allocateArray(Memory.LONG, values));
                }
 
                public final MemoryAddress address() {
@@ -401,16 +425,24 @@ public class Memory {
        public static class FloatArray extends AbstractList<Float> implements Memory.Addressable {
                final MemorySegment segment;
 
-               public FloatArray(MemorySegment segment) {
+               private FloatArray(MemorySegment segment) {
                        this.segment = segment;
                }
 
-               public FloatArray(Frame frame, long size) {
-                       this(frame.allocateArray(Memory.FLOAT, size));
+               public static FloatArray create(MemorySegment segment) {
+                       return new FloatArray(segment);
+               }
+
+               public static FloatArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+                       return create(MemorySegment.ofAddress(address, length, scope));
+               }
+
+               public static FloatArray createArray(long length, SegmentAllocator alloc) {
+                       return create(alloc.allocateArray(Memory.FLOAT, length));
                }
 
-               public FloatArray(Frame frame, float... values) {
-                       this(frame.allocateArray(Memory.FLOAT, values));
+               public static FloatArray create(SegmentAllocator alloc, float... values) {
+                       return create(alloc.allocateArray(Memory.FLOAT, values));
                }
 
                public final MemoryAddress address() {
@@ -454,16 +486,24 @@ public class Memory {
        public static class DoubleArray extends AbstractList<Double> implements Memory.Addressable {
                final MemorySegment segment;
 
-               public DoubleArray(MemorySegment segment) {
+               private DoubleArray(MemorySegment segment) {
                        this.segment = segment;
                }
 
-               public DoubleArray(Frame frame, long size) {
-                       this(frame.allocateArray(Memory.DOUBLE, size));
+               public static DoubleArray create(MemorySegment segment) {
+                       return new DoubleArray(segment);
+               }
+
+               public static DoubleArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+                       return create(MemorySegment.ofAddress(address, length, scope));
                }
 
-               public DoubleArray(Frame frame, double... values) {
-                       this(frame.allocateArray(Memory.DOUBLE, values));
+               public static DoubleArray createArray(long length, SegmentAllocator alloc) {
+                       return create(alloc.allocateArray(Memory.DOUBLE, length));
+               }
+
+               public static DoubleArray create(SegmentAllocator alloc, double... values) {
+                       return create(alloc.allocateArray(Memory.DOUBLE, values));
                }
 
                public final MemoryAddress address() {
@@ -515,8 +555,16 @@ public class Memory {
                        return new PointerArray(segment);
                }
 
-               public static PointerArray createArray(long size, SegmentAllocator alloc) {
-                       return create(alloc.allocateArray(Memory.POINTER, size));
+               public static PointerArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+                       return create(MemorySegment.ofAddress(address, length, scope));
+               }
+
+               public static PointerArray createArray(long length, SegmentAllocator alloc) {
+                       return create(alloc.allocateArray(Memory.POINTER, length));
+               }
+
+               public static PointerArray create(Frame alloc, MemoryAddress... values) {
+                       return create(alloc.allocateArray(Memory.POINTER, values));
                }
 
                public final MemoryAddress address() {
@@ -557,23 +605,38 @@ public class Memory {
                }
        }
 
+       // This needs a separate scope from the array itself
        public static class HandleArray<T extends Memory.Addressable> extends AbstractList<T> implements Memory.Addressable {
                final MemorySegment segment;
+               final ResourceScope scope;
                BiFunction<MemoryAddress,ResourceScope,T> create;
 
-               private HandleArray(MemorySegment segment, BiFunction<MemoryAddress,ResourceScope,T> create) {
+               private HandleArray(MemorySegment segment, BiFunction<MemoryAddress,ResourceScope,T> create, ResourceScope scope) {
                        this.segment = segment;
                        this.create = create;
+                       this.scope = scope;
                }
 
                public static <T extends Memory.Addressable> HandleArray<T> create(MemorySegment segment, BiFunction<MemoryAddress,ResourceScope,T> create) {
-                       return new HandleArray<>(segment, create);
+                       return new HandleArray<>(segment, create, segment.scope());
+               }
+
+               public static <T extends Memory.Addressable> HandleArray<T> create(MemorySegment segment, BiFunction<MemoryAddress,ResourceScope,T> create, ResourceScope scope) {
+                       return new HandleArray<>(segment, create, scope);
                }
 
                public static <T extends Memory.Addressable> HandleArray<T> createArray(long size, SegmentAllocator alloc, BiFunction<MemoryAddress,ResourceScope,T> create) {
                        return create(alloc.allocateArray(Memory.POINTER, size), create);
                }
 
+               public static <T extends Memory.Addressable> HandleArray<T> createArray(long size, SegmentAllocator alloc, BiFunction<MemoryAddress,ResourceScope,T> create, ResourceScope scope) {
+                       return create(alloc.allocateArray(Memory.POINTER, size), create, scope);
+               }
+
+               public static <T extends Memory.Addressable> HandleArray<T> createArray(MemoryAddress address, long size, BiFunction<MemoryAddress,ResourceScope,T> create, ResourceScope scope) {
+                       return create(MemorySegment.ofAddress(address, size * Memory.POINTER.byteSize(), scope), create);
+               }
+
                @Override
                public final MemoryAddress address() {
                        return segment.address();
index a458279..a7a8666 100644 (file)
@@ -13,17 +13,23 @@ api_demo_SOURCES := $(wildcard src/api/test/*.java)
 all::
        mkdir -p bin
 
-all:: bin/demo.built
+all:: bin/demo.built bin/libapi.so
 
 bin/api.built: bin/api.gen
        $(JAVAC) $(JAVACFLAGS) -cp bin/classes -d bin/classes \
                $(shell find bin/java -name '*.java')
        touch $@
 
-bin/api.gen: bin/api.pm ../src/generate-native $(api_SOURCES)
-       ../src/generate-native -d bin/java -t proto.api ./bin/api.pm
+bin/api.gen: bin/api.pm ../src/generate-native $(api_SOURCES)  bin/api-defines.pm api.api
+       ../src/generate-native -v -d bin/java -t proto.api -a ./bin/api.pm -a ./bin/api-defines.pm api.api
        touch $@
 
+bin/api-defines.pm: api.h ../src/export-defines api.api
+       ../src/export-defines -d bin/api-defines.c api.api
+       $(CC) -o bin/api-defines -I. bin/api-defines.c
+       bin/api-defines $@~
+       mv $@~ $@
+
 bin/api.pm: api.h ../src/export.so
        gcc -fplugin=../src/export.so -fplugin-arg-export-output=$@~ ./$< -o /dev/null
        mv $@~ $@
diff --git a/test-api/README b/test-api/README
new file mode 100644 (file)
index 0000000..59a3438
--- /dev/null
@@ -0,0 +1,8 @@
+
+Introduction
+------------
+
+Small generator and prototyping area for a c library.
+
+This is mostly to test out ideas like pattern matching for mostly
+automatic api conversion.
diff --git a/test-api/api.api b/test-api/api.api
new file mode 100644 (file)
index 0000000..cfe3103
--- /dev/null
@@ -0,0 +1,15 @@
+
+# collect all functions and the defines we want
+library APILib load=api {
+       define:api-defines
+       func://
+}
+
+# grab all structures
+struct // default=all field:rename=studly-caps {
+}
+
+# grab all defines in api.h
+define api-defines api.h {
+       api.h       file-include
+}
index 0aad6b8..58cac5f 100644 (file)
@@ -1,6 +1,7 @@
 
 #include <stdio.h>
 #include <string.h>
+#include <stdlib.h>
 #include "api.h"
 
 static void funca(int a) {
index e4639ca..4e6b023 100644 (file)
@@ -1,4 +1,13 @@
 
+#define API_FOO "12"
+#define API_MAKEVERSION(a, b) (a <<16) | b
+#define API_VERSION API_MAKEVERSION(1, 0)
+#define API_A 12UL
+#define API_B 12L
+#define API_C 12U
+#define API_D 12.0
+#define API_E 12.0f
+
 struct data {
        struct data *next;
 
@@ -22,3 +31,9 @@ struct api {
 
 void *api_func(const char *name);
 struct api *api_create(void);
+
+struct list_node {
+       struct list_node *succ;
+       struct list_node *pred;
+       char name[64-16];
+};
index 2d8be3b..108d898 100644 (file)
@@ -35,7 +35,7 @@ public class TestAPI {
                        b.setC((byte)255);
                        b.setD((byte)255);
 
-                       b.setTest_a(cb);
+                       b.setTestA(cb);
 
                        System.out.println("from a");
                        print_data(a);
@@ -49,7 +49,7 @@ public class TestAPI {
 
                        // dynamic lookup
                        System.out.println("call funca via symbol lookup");
-                       Memory.FunctionPointer<Call_i32_v> funca = Call_i32_v.downcall(api_func(Memory.ByteArray.create("funca", frame)), scope);
+                       Memory.FunctionPointer<Call_i32_v> funca = Call_i32_v.downcall(api_func("funca"), scope);
                        System.out.printf(" %s\n", funca.symbol());
                        funca.function().call(12);
 
diff --git a/test-ffmpeg/Makefile b/test-ffmpeg/Makefile
new file mode 100644 (file)
index 0000000..e1403fc
--- /dev/null
@@ -0,0 +1,52 @@
+
+CFLAGS=-g -fPIC
+HOST_CC=gcc
+
+JAVA_HOME=/opt/jdk-foreign/jvm/openjdk-19-internal
+JAVAC=$(JAVA_HOME)/bin/javac
+JAVA=$(JAVA_HOME)/bin/java
+
+JAVACFLAGS=--add-modules jdk.incubator.foreign
+
+api_SOURCES := $(wildcard ../src/api/*.java)
+ffmpeg_SOURCES := $(wildcard src/proto/ffmpeg/*.java)
+ffmpeg_demo_SOURCES := $(wildcard src/ffmpeg/test/*.java)
+
+all::
+       mkdir -p bin
+
+all:: bin/demo.built
+
+bin/ffmpeg.built: bin/ffmpeg.gen $(ffmpeg_SOURCES)
+       $(JAVAC) $(JAVACFLAGS) -cp bin/classes -d bin/classes \
+               $(shell find bin/java -name '*.java') \
+               $(ffmpeg_SOURCES)
+       touch $@
+
+bin/ffmpeg.gen: bin/ffmpeg.pm bin/ffmpeg-defines.pm ../src/generate-native $(api_SOURCES)
+       ../src/generate-native -d bin/java -t proto.ffmpeg -a ./bin/ffmpeg.pm -a ./bin/ffmpeg-defines.pm ffmpeg.api
+       touch $@
+
+bin/ffmpeg-defines.pm: ffmpeg.h ../src/export-defines ffmpeg.api
+       ../src/export-defines -d bin/ffmpeg-defines.c ffmpeg.api
+       $(HOST_CC) -o bin/ffmpeg-defines -I. bin/ffmpeg-defines.c
+       bin/ffmpeg-defines $@~
+       mv $@~ $@
+
+bin/ffmpeg.pm: ffmpeg.h ../src/export.so
+       gcc -fplugin=../src/export.so -fplugin-arg-export-output=$@~ ./$< -o /dev/null
+       mv $@~ $@
+
+bin/demo.built: $(ffmpeg_demo_SOURCES) bin/ffmpeg.built
+       $(JAVAC) $(JAVACFLAGS) -cp bin/classes -d bin/classes $(ffmpeg_demo_SOURCES)
+       touch $@
+
+demo: all
+       $(JAVA) --enable-native-access=ALL-UNNAMED --add-modules jdk.incubator.foreign \
+               -cp bin/classes \
+               ffmpeg.test.TestFFMPEG
+
+clean:
+       rm -rf bin 000[0-9].pnm
+
+.PHONY: demo clean all
diff --git a/test-ffmpeg/README b/test-ffmpeg/README
new file mode 100644 (file)
index 0000000..e22691f
--- /dev/null
@@ -0,0 +1,12 @@
+
+Introduction
+------------
+
+Small demonstrator of generate-native on a complex and messy api
+wrapped in a Java "object oriented" one.
+
+make demo
+---------
+
+Copy some video file to 'movie.avi' to run the demo.  It will output
+the first 10 frames to "\d{5}.pnm".
diff --git a/test-ffmpeg/ffmpeg.api b/test-ffmpeg/ffmpeg.api
new file mode 100644 (file)
index 0000000..777d287
--- /dev/null
@@ -0,0 +1,222 @@
+
+# access=rwi
+#  r read (get)
+#  w write (set)
+#  i read/write indexed (get/setAtIndex)
+
+struct <default> field:rename=studly-caps func:rename=camel-case access=rw {
+}
+
+struct AVFormatContext default=none func:rename=s/^(avformat_|av_)//,camel-case field:rename=studly-caps {
+       iformat         access=r rename=InputFormat
+       oformat         access=r rename=OutputFormat
+       pb                       rename=IOContext
+
+       ctx_flags       access=r
+       nb_streams      access=r rename=NumStreams
+
+       streams         array-size=nb_streams
+
+       start_time      access=r
+       duration        access=r
+       bit_rate        access=r
+
+       interrupt_callback
+
+       func:avformat_open_input        static constructor=byref constructor-result=0 success=0
+       func:avformat_find_stream_info  rename=findStreamInfo instance=0
+
+       func:av_read_frame
+}
+
+struct AVStream default=none field:rename=studly-caps {
+       index           access=r
+       id              access=r rename=ID
+       time_base
+       start_time
+       duration
+       nb_frames       rename=NumFrames
+       discard
+       avg_frame_rate  rename=AverageFrameRate
+       sample_aspect_ratio
+       codecpar        rename=CodecParameters
+}
+
+struct AVCodec access=r default=none func:rename=s/^avcodec_//,camel-case field:rename=studly-caps {
+       id                       rename=ID
+       name
+       long_name
+       type
+       capabilities
+       max_lowres
+       # need some sort of length=null-terminated here i suppose
+       supported_framerates    rename=framerates
+       pix_fmts                        rename=pixelFormats
+
+       func:av_codec_next                      static rename=next
+       func:avcodec_find_decoder               static
+       func:avcodec_find_decoder_by_name       static
+       # here or on AVCodecContext?
+       func:avcodec_alloc_context3             rename=allocContext
+
+       define:AVCodecBits
+}
+
+struct AVCodecContext default=none func:rename=s/^avcodec_//,camel-case field:rename=studly-caps {
+       codec_id      rename=CodecID
+
+       skip_loop_filter
+       skip_idct
+       skip_frame
+
+       func:avcodec_alloc_context3     static rename=alloc
+       func:avcodec_open2              rename=open
+       func:avcodec_send_packet
+       func:avcodec_receive_packet
+       func:avcodec_send_frame
+       func:avcodec_receive_frame
+
+       define:AVCodecContextBits
+}
+
+struct AVFrame default=all func:rename=s/^av_frame_//,camel-case field:rename=studly-caps {
+       func:av_frame_alloc  static
+       func:av_frame_free
+}
+
+struct AVFrameSideData default=none {
+}
+
+struct AVRational field:rename=studly-caps {
+}
+
+struct AVCodecParameters func:rename=s/^avcodec_parameters_//,camel-case field:rename=studly-caps {
+       codec_id                 rename=CodecID
+       func:avcodec_parameters_alloc           static
+       func:avcodec_parameters_free
+       func:avcodec_parameters_copy
+       func:avcodec_parameters_to_context      instance=1
+       func:avcodec_parameters_from_context
+}
+
+struct AVPacket default=all func:rename=s/^av_packet_//,camel-case field:rename=studly-caps {
+       pts|dts    rename=upper-leadin
+       data       array-size=size
+
+       func:av_packet_alloc  static
+       func:av_packet_free
+       func:av_init_packet   rename=init
+}
+
+struct SwsContext func:rename=s/^sws_// {
+#       func:/^sws_/
+       func:sws_getContext     static
+       func:sws_freeContext
+       func:sws_scale
+
+       define:sws
+}
+
+struct SwsFilter {
+}
+struct SwsVector {
+}
+struct AVPixFmtDescriptor {
+}
+struct AVComponentDescriptor access=rwi {
+}
+
+library AVUtil {
+       func:/^av_image_/
+}
+
+struct AVProbeData default=none field:rename=studly-caps {
+}
+
+struct AVBufferRef access=rwi default=none {
+}
+struct AVPacketSideData default=none {
+}
+struct AVIOContext default=none {
+}
+struct AVIOInterruptCB default=none {
+}
+
+struct AVDictionaryEntry {
+}
+
+struct AVDictionary func:rename=s/^av_dict_//,camel-case {
+       define:dict
+       func:/av_dict_/
+}
+
+struct AVInputFormat default=none access=r field:rename=studly-caps {
+       name
+       long_name
+       mime_type
+       flags
+       extensions
+# want to define functions here
+       func:av_iformat_next          static     rename=next
+       func:av_register_input_format instance=0 rename=register
+}
+
+struct AVOutputFormat default=none field:rename=studly-caps {
+       name
+       long_name
+       mime_type
+       flags
+       extensions
+       audio_codec
+       video_codec
+       subtitle_codec
+}
+
+define dict ffmpeg.h {
+       /AV_DICT_/
+}
+
+define AVChannelLayoutBits ffmpeg.h {
+       /^AV_CH_LAYOUT_/        x64
+}
+
+define AVErrorBits ffmpeg.h {
+       /^AVERROR_/             x32
+}
+
+define AVCodecContextBits ffmpeg.h {
+       /^AV_CODEC_FLAG_|AV_CODEC_FLAG2_/       u32
+       /^AV_INPUT_BUFFER_/     i32
+}
+
+define AVCodecBits ffmpeg.h {
+       /^AV_CODEC_CAP_/        x32
+}
+
+define AVIOContextBits ffmpeg.h {
+       /^AVSEEK_|AVIO_FLAG_|AVIO_SEEKABLE_/
+}
+
+define AVOptionsBits ffmpeg.h {
+       /^AV_OPT_/
+}
+
+define sws ffmpeg.h {
+       /^SWS_/
+}
+
+# This is used to define 'char *' arguments which are not strings.
+#  ... a bit clumsy
+
+func av_image_copy_plane {
+       src|dst          array
+}
+func av_image_copy_to_buffer {
+       dst array
+}
+func av_image_fill_arrays {
+       src array
+}
+func av_image_fill_pointers {
+       ptr array
+}
diff --git a/test-ffmpeg/ffmpeg.h b/test-ffmpeg/ffmpeg.h
new file mode 100644 (file)
index 0000000..1d3a58e
--- /dev/null
@@ -0,0 +1,6 @@
+
+#include <libavcodec/avcodec.h>
+#include <libavformat/avformat.h>
+#include <libswscale/swscale.h>
+#include <libavutil/imgutils.h>
+#include <libavutil/opt.h>
diff --git a/test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java b/test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java
new file mode 100644 (file)
index 0000000..cbec649
--- /dev/null
@@ -0,0 +1,137 @@
+
+package ffmpeg.test;
+
+import jdk.incubator.foreign.*;
+import java.io.FileOutputStream;
+import java.io.IOException;
+
+import proto.ffmpeg.Memory.*;
+import proto.ffmpeg.*;
+
+import static proto.ffmpeg.AVMediaType.*;
+import static proto.ffmpeg.AVPixelFormat.*;
+
+public class TestFFMPEG {
+
+       public static void main(String[] args) throws Exception {
+               System.loadLibrary("avcodec");
+               System.loadLibrary("avformat");
+               System.loadLibrary("swscale");
+               System.loadLibrary("avutil");
+
+               try (Frame frame = Memory.createFrame();
+                       ResourceScope scope = ResourceScope.newConfinedScope()) {
+                       int res;
+
+                       if (false) {
+                               AVInputFormat avif = null;
+                               while ((avif = AVInputFormat.next(avif)) != null) {
+                                       System.out.printf("name: %30s %s\n",
+                                               avif.getName(),
+                                               avif.getLongName());
+                               }
+                       }
+
+                       AVFormatContext format = AVFormatContext.openInput("movie.avi", null, null, scope);
+
+                       if (format == null) {
+                               throw new IOException("File not found: movie.avi");
+                       }
+
+                       format.findStreamInfo(null);
+
+                       HandleArray<AVStream> streams = format.getStreams();
+
+                       System.out.printf(" streams=%d %d\n", format.getNumStreams(), streams.size());
+
+                       for (AVStream s: streams) {
+                               AVCodecParameters cp = s.getCodecParameters();
+
+                               System.out.printf(" stream [index=%d, id=%d]\n", s.getIndex(), s.getID());
+                               System.out.printf("  start=%d\n  duration=%d\n", s.getStartTime(), s.getDuration());
+
+                               System.out.printf("  codecType: %d\n", cp.getCodecType());
+                               System.out.printf("  codecId: %d\n", cp.getCodecID());
+                               System.out.printf("  bitrate: %d\n", cp.getBitRate());
+                               System.out.printf("  samplerate: %d\n", cp.getSampleRate());
+                       }
+
+                       AVCodecParameters vcp = null;
+                       AVStream vstream = null;
+
+                       for (AVStream stream: streams) {
+                               AVCodecParameters cp = stream.getCodecParameters();
+
+                               switch (cp.getCodecType()) {
+                               case AVMEDIA_TYPE_VIDEO:
+                                       System.out.printf("  video = %d\n", stream.getIndex());
+                                       vstream = stream;
+                                       vcp = cp;
+                                       break;
+                               case AVMEDIA_TYPE_AUDIO:
+                                       System.out.printf("  audio = %d\n", stream.getIndex());
+                                       break;
+                               }
+                       }
+
+                       AVCodec codec = AVCodec.findDecoder(vcp.getCodecID());
+                       //AVCodecContext c = AVCodecContext.alloc(codec);
+                       AVCodecContext c = codec.allocContext();
+                       AVFrame avframe = AVFrame.alloc();
+                       AVPacket packet = AVPacket.alloc();
+
+                       packet.init();
+                       vcp.toContext(c);
+                       c.setCodecID(codec.getID());
+                       res = c.open(codec, null);
+
+                       System.out.printf("c.open = %d\n", res);
+
+                       AVPixelReader pix = null;
+                       long nframes = 0;
+                       long pts = 0;
+
+                       while ((res = format.readFrame(packet)) >= 0) {
+                               int index = packet.getStreamIndex();
+
+                               if (index == vstream.getIndex()) {
+                                       //boolean ready;
+                                       int ready;
+
+                                       pts = packet.getDTS();
+
+                                       res = c.sendPacket(packet);
+                                       res = c.receiveFrame(avframe);
+
+                                       if (res == 0) {
+                                               int w = avframe.getWidth();
+                                               int h = avframe.getHeight();
+                                               int fmt = avframe.getFormat();
+
+                                               if (true) {
+                                                       if (pix == null) {
+                                                               //pix = frame.getPixelReader(w, h, 0);
+                                                               pix = new FramePixelReader(avframe, w, h, 0);
+                                                       }
+
+                                                       byte[] pixels = new byte[w*h*3];
+
+                                                       pix.getPixels(0, h, AV_PIX_FMT_RGB24, pixels, 0, 0);
+
+                                                       try (FileOutputStream fos = new FileOutputStream(String.format("%04d.pnm", nframes))) {
+                                                               String header = String.format("P6\n%d %d\n255\n", w, h);
+                                                               fos.write(header.getBytes());
+                                                               fos.write(pixels);
+                                                       }
+                                               }
+
+                                               System.out.printf("%06d video frame %dx%d @ %d\n", nframes, w, h, fmt);
+                                               nframes++;
+                                       }
+                               }
+                               if (nframes == 10)
+                                       break;
+                       }
+               }
+       }
+}
diff --git a/test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java b/test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java
new file mode 100644 (file)
index 0000000..6ca4a3c
--- /dev/null
@@ -0,0 +1,53 @@
+/*
+ * Copyright (C) 2017 Michael Zucchi
+ *
+ * This file is part of jjmpeg <https://www.zedzone.space/software/jjmpeg.html>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package proto.ffmpeg;
+
+import java.nio.ByteBuffer;
+
+/**
+ * AVPixelReader is a high level interface to SwsContext.
+ * <p>
+ * This interface allows one to retrieve pixels by bands to a specific format.
+ *
+ * @see AVFrame#getPixelReader
+ */
+public interface AVPixelReader {
+
+       /**
+        * Read pixels into a Buffer. The buffer must be a Direct*Buffer.
+        *
+        * @param <T>
+        * @param y Start scanline. Currently only 0 works.
+        * @param h End scanline output. Currently only the output height works.
+        * @param fmt Pixel format to write to.
+        * @param buffer Target for pixels.
+        * @param scanlineStride Should match the output width and may be ignored.
+        */
+       public void getPixels(int y, int h, int fmt, ByteBuffer buffer, int scanlineStride);
+
+       public void getPixels(int y, int h, int fmt, byte[] buffer, int offset, int scanlineStride);
+
+       public void getPixels(int y, int h, int fmt, short[] buffer, int offset, int scanlineStride);
+
+       public void getPixels(int y, int h, int fmt, int[] buffer, int offset, int scanlineStride);
+
+       public long bufferSize(int h, int fmt);
+
+       public void release();
+}
diff --git a/test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java b/test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java
new file mode 100644 (file)
index 0000000..26ec177
--- /dev/null
@@ -0,0 +1,153 @@
+/*
+ * Copyright (C) 2020 Michael Zucchi
+ *
+ * This file is part of jjmpeg <https://www.zedzone.space/software/jjmpeg.html>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package proto.ffmpeg;
+
+import java.nio.Buffer;
+import java.nio.ByteBuffer;
+import java.nio.BufferOverflowException;
+import jdk.incubator.foreign.*;
+import proto.ffmpeg.Memory.*;
+
+public class FramePixelReader implements AVPixelReader {
+
+       SwsContext sws;
+       final AVFrame frame;
+       final int dwidth, dheight, flags;
+       int dfmt = -1;
+
+       ResourceScope scope = ResourceScope.newImplicitScope();
+       MemorySegment seg;
+
+       public FramePixelReader(AVFrame frame, int dwidth, int dheight, int flags) {
+               this.frame = frame;
+               this.dwidth = dwidth;
+               this.dheight = dheight;
+               this.flags = flags;
+       }
+
+       SwsContext getContext(int fmt) {
+               if (sws == null || fmt != dfmt) {
+                       dfmt = fmt;
+                       if (sws != null) {
+                               //sws.release();
+                       }
+                       sws = SwsContext.getContext(frame.getWidth(), frame.getHeight(), frame.getFormat(),
+                               dwidth, dheight, dfmt, flags,
+                               null, null, null);
+               }
+               return sws;
+       }
+
+       MemorySegment getSegment(long size) {
+               if (seg == null || seg.byteSize() < size) {
+                       if (seg != null) {
+                               //Memory.free(seg.baseAddress());
+                       }
+                       //seg = Memory.ofNative(Memory.malloc(size), size);
+                       seg = MemorySegment.allocateNative(size, 16, scope);
+               }
+               return seg;
+       }
+
+       private void getPixels(int y, int h, int fmt, MemorySegment buffer, int scanlineStride) {
+               SwsContext ctx = getContext(fmt);
+
+               // FIXME: check sizes
+
+               try (Frame a = Memory.createFrame()) {
+                       IntArray stride = IntArray.createArray(4, a);
+                       PointerArray data = PointerArray.createArray(4, a);
+                       ByteArray buf = ByteArray.create(buffer);
+                       int res;
+
+                       res = AVUtil.av_image_fill_linesizes(stride, fmt, dwidth);
+                       res = AVUtil.av_image_fill_pointers(data, fmt, h, buf, stride);
+
+                       System.out.printf(" base     %016x\n", buffer.address().toRawLongValue());
+                       for (int i=0;i<4;i++) {
+                               System.out.printf(" %d %6d %016x\n", i, stride.get(i), data.get(i).address().toRawLongValue());
+                       }
+
+                       if (res > 0 && res <= buffer.byteSize()) {
+                               ctx.scale(
+                                       frame.getData(),
+                                       frame.getLinesize(),
+                                       y, h,
+                                       data, stride);
+                       } else {
+                               throw new BufferOverflowException();
+                       }
+               }
+       }
+
+       public long bufferSize(int h, int fmt) {
+               try (Frame a = Memory.createFrame()) {
+                       IntArray stride = IntArray.createArray(4, a);
+                       PointerArray data = PointerArray.createArray(4, a);
+                       int res;
+
+                       res = AVUtil.av_image_fill_linesizes(stride, fmt, dwidth);
+                       res = AVUtil.av_image_fill_pointers(data, fmt, h, null, stride);
+
+                       return res;
+               }
+       }
+
+       public void getPixels(int y, int h, int fmt, ByteBuffer buffer, int scanlineStride) {
+               getPixels(y, h, fmt, MemorySegment.ofByteBuffer((ByteBuffer)buffer), scanlineStride);
+       }
+
+       public void getPixels(int y, int h, int fmt, byte[] buffer, int offset, int scanlineStride) {
+               SwsContext ctx = getContext(fmt);
+
+               try (Frame a = Memory.createFrame()) {
+                       IntArray stride = IntArray.createArray(4, a);
+                       PointerArray data = PointerArray.createArray(4, a);
+                       MemorySegment seg = getSegment(bufferSize(h, fmt));
+                       ByteArray dst = ByteArray.create(seg);
+                       int res;
+
+                       res = AVUtil.av_image_fill_linesizes(stride, fmt, dwidth);
+                       res = AVUtil.av_image_fill_pointers(data, fmt, h, dst, stride);
+
+                       AVUtil.av_image_fill_linesizes(stride, fmt, dwidth);
+
+                       ctx.scale(
+                               frame.getData(),
+                               frame.getLinesize(),
+                               y, h,
+                               data, stride);
+
+                       MemorySegment.ofArray(buffer).copyFrom(seg);
+               }
+       }
+
+       public void getPixels(int y, int h, int fmt, short[] buffer, int offset, int scanlineStride) {
+       }
+
+       public void getPixels(int y, int h, int fmt, int[] buffer, int offset, int scanlineStride) {
+       }
+
+       public void release() {
+               if (scope != null) {
+                       scope.close();
+                       scope = null;
+               }
+       }
+}