From 6e4db7e40a15cc8164c2cce949ffaf656f473426 Mon Sep 17 00:00:00 2001 From: Not Zed Date: Wed, 5 Jan 2022 09:19:15 +1030 Subject: [PATCH] Pattern matching and defaults for structs Added ffmpeg example. Changed test-api to be c-like example. --- README | 105 +- src/export-defines | 394 ++++-- src/export.cc | 8 +- src/generate-native | 1100 ++++++++++++++--- src/template/Memory.java | 121 +- test-api/Makefile | 12 +- test-api/README | 8 + test-api/api.api | 15 + test-api/api.c | 1 + test-api/api.h | 15 + test-api/src/api/test/TestAPI.java | 4 +- test-ffmpeg/Makefile | 52 + test-ffmpeg/README | 12 + test-ffmpeg/ffmpeg.api | 222 ++++ test-ffmpeg/ffmpeg.h | 6 + test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java | 137 ++ .../src/proto/ffmpeg/AVPixelReader.java | 53 + .../src/proto/ffmpeg/FramePixelReader.java | 153 +++ 18 files changed, 2052 insertions(+), 366 deletions(-) create mode 100644 test-api/README create mode 100644 test-api/api.api create mode 100644 test-ffmpeg/Makefile create mode 100644 test-ffmpeg/README create mode 100644 test-ffmpeg/ffmpeg.api create mode 100644 test-ffmpeg/ffmpeg.h create mode 100644 test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java create mode 100644 test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java create mode 100644 test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java diff --git a/README b/README index 6fe7c06..cdb34e6 100644 --- a/README +++ b/README @@ -2,91 +2,53 @@ Introduction ------------ -This is an alternative to jextract for the OpenJDK panama project -which in turn is a project to implement Java to Native interfaces at -runtime. +This is some experiments with the panama-foreign abi JEP for invoking +native C functions from Java directly and without JNI. -It uses a gcc plugin for type extraction and a "simple" perl script to -transform this into Java source files with the appropriate annotations -for panama to do it's thing. +The main goal is to experiment with creating a "java friendly" and +mostly type-safe api directly in one step, without requiring +additional wrapping or inconvenient use. -Compile -------- - -Edit the makefile to point to your gcc directory and make. - -$ cd src -$ vi Makefile -$ make - -generate is a perl script. - -See test-api/* for a simple example. JAVA_HOME must point to -a compatible panama-enabled jdk. - -Use dump.so ------------ - -dump.so is a gcc plugin which captures structures, unions, enums, -exported functions, and a few typedefs into a perl source file -containing a single hash. - -Put all the include files required for your api in api.h. - -$ gcc -fplugin=src/export.so -fplugin-arg-export-output=api.pm api.h -o /dev/null +It's roughly tracking the openjdk repository of the same date. -api.pm is a text file of perl source-code which contains a descriptor -of all supported types found. +It uses a gcc plugin to compile the headers to obtain most of the api +information, but requires cpp and perl to extract the values of +#define constants. -Use generate ------------- +This api information is then converted to Java wrappers using a +config-directed perl script `generate-native'. The config is flexible +enough to either generate c-like libraries of static functions or +object-oriented layouts with data, static and object functions grouped +by class. -Quick and dirty: +test-vulkan uses a different approach as the vulkan api is officially +defined by an xml registry and not the generated C headers. This is +directly converted to about-as-java-friendly a vulkan api as one can +hope for (all those structs you need to initialise get messy in any +language). Most of the script is converting the registry vk.xml file +into a usable structure as it's really only designed for outputting c +headers. -$ generate -d dir -t name.space -c APILib -lapi ./api.pm +Compile +------- -Generate all structures in api.pm (the ./ is required) and place all -functions into dir/name/space/APILib.java which will link with -libapi.so. +Requirements are gcc and cpp, perl, GNU make. -The generator is quite flexible, sub-sets of functions can be placed -in alternative *Lib.java files, types can be filtered. If a function -or type is specified explicitly, then only those types it requires are -included in the generated output (recursively). +First run make in src. -See the top of the file for details, it's probably up to date. +See test-*/* for examples. JAVA_HOME must point to a compatible +panama-enabled jdk. Status ------ -This is currently just a 'quick and dirty' weekend hack. The tools -spew a lot of debug. dump.cc is basic c-compatible-c++, because I -don't know c++. - -dump.so works by hooking into the gcc pre-compiled headers function, -it might work with a c source file as well. It's only been used with -gcc 9. - -The dumper will hard-fail if it sees defintions it doesn't understand, -and currently it's only been tested with a small nubmer of libraries. - -Enumerations are currently translated to integers. The generator -doesn't currently export any of the symbolic values to Java but it is -easy to add. - -varags is not implemented. - -All function pointers are translated to a unique set of functional -interfaces whose names are based on the function signature. So the -type names are ugly but they are stable. - -Anonymous type names are based on the structure member they belong to -so they may not be unique. - -It will only ever support c. +It's all work in progress of course. -Only tested against the panama 'openjdk-14-panama+1-15_linux' on -slackware-current @ 2020.01.01 and gcc 9.2.0. +* bitfields not implemented yet. +* varargs is not implemented. +* the support library in Memory.java is copied to each output api, but should + be global. +* the scope and object lifecycle stuff is not really sorted out yet. License ------- @@ -99,3 +61,4 @@ Links * https://www.zedzone/software/panamaz.html - project page. * https://openjdk.java.net/projects/panama - openjdk panama page. + * https://github.com/openjdk/panama-foreign - openjdk panama source. diff --git a/src/export-defines b/src/export-defines index ece2b0e..79b35e5 100755 --- a/src/export-defines +++ b/src/export-defines @@ -1,91 +1,345 @@ #!/usr/bin/perl -# the gcc plugin doesn't have access to the #defines, so this is a 'simple' way to get them. +use Data::Dumper; -@matchDefine = (); -@matchInclude = (); - -@includes = (); +my @includes = (); +my $header; +my $control = "api-defines.def"; +my $output; while (@ARGV) { - my $cmd = shift(@ARGV); - - if ($cmd eq "-d") { - my $v = shift(@ARGV); - push @matchDefine, $v; - } elsif ($cmd eq "--define-file") { - my $file = shift(@ARGV); - push @matchDefine, readMatchFile($file); - } elsif ($cmd eq "-i") { - my $v = shift(@ARGV); - push @matchInclude, $v; - } elsif ($cmd eq "-I") { - my $v = shift(@ARGV); - push @includes, $v; - } elsif ($cmd =~ "-I(.*)") { - push @includes, $1; - } else { - $header = $cmd; - } + my $cmd = shift; + + if ($cmd eq "-t") { + $package = shift; + } elsif ($cmd eq "-d") { + $output = shift; + } elsif ($cmd eq "-v") { + $verbose++; + } elsif ($cmd eq "-I") { + push @includes, shift; + } else { + $control = $cmd; + } +} + +die ("no output specified") if !$output; + +my $defs = loadControlFile($control); +my @exports = @{$defs->{define}}; +my %rawDefines = (); # indexed by header + +my $CPPFLAGS = ""; + +foreach $i (@includes) { + $CPPFLAGS .= " '-I$i'"; +} + +# flatten the generic format to a more usable format, work out defaults (. and syntax check?) +foreach $export (@exports) { + my $includes = 0, $excludes = 0; + + foreach $inc (@{$export->{items}}) { + my @options = @{$inc->{options}}; + + if ($inc->{match} =~ m@^/(.*)/$@) { + $inc->{regex} = qr/$1/; + } else { + $inc->{regex} = qr/^$inc->{match}$/; + } + + $inc->{mode} = "include"; + foreach $o (@{$inc->{options}}) { + if ($o =~ m/^(exclude|include|file-include|file-exclude)$/) { + $inc->{mode} = $o; + } + } + if ($inc->{mode} =~ m/include/) { + $includes += 1; + } else { + $excludes += 1; + } + } + + $export->{default} = 'all'; + $export->{default} = 'none' if ($includes > 0); + + my @options = @{$export->{options}}; + + foreach $o (@options) { + if ($o =~ m/header=(.*)/) { + $export->{header} = $1; + } elsif ($o =~ m/default=(.*)/) { + $export->{default} = $1; + } elsif ($#options == 0) { + $export->{header} = $o; + } else { + print STDERR "unknown defines option '$o'\n"; + } + } + + # insert ignore thing + if ($export->{default} eq 'all') { + unshift @{$export->{items}}, { regex => qr//, mode => 'file-exclude' }; + } + + die ("no header for '$export->{name}'") if !$export->{header}; +} + +# load all defines once and link in +# keep_comments is a bit broken +foreach $export (@exports) { + my $header = $export->{header}; + + if (!defined($rawDefines{$header})) { + $rawDefines{$header} = scanDefines($header, { CPPFLAGS=>$cppflags, keep_comments=>0 }); + } + + $export->{rawDefines} = $rawDefines{$header}; +} + +foreach $export (@exports) { + # extract matching #defines + my @defines = (); + + foreach $d (@{$export->{rawDefines}}) { + my $output = 0; + my $matches = 0; + + print "? $d->{name} $d->{file}" if $verbose; + foreach $inc (@{$export->{items}}) { + if ($inc->{mode} eq "include") { + $matches = $d->{name} =~ m/$inc->{regex}/; + $output = 1 if $matches; + } elsif ($inc->{mode} eq "exclude") { + $matches = $d->{name} =~ m/$inc->{regex}/; + } elsif ($inc->{mode} eq "file-include") { + $matches = $d->{file} =~ m/$inc->{regex}/; + $output = 1 if $matches; + } elsif ($inc->{mode} eq "file-exclude") { + $matches = $d->{file} =~ m/$inc->{regex}/; + } + print " ($inc->{mode} '$inc->{match}' =$matches)" if $verbose; + last if $matches; + } + + $output = 1 if (!$matches && $export->{default} eq "all"); + + print " output=$output\n" if $verbose; + + push (@{$export->{export}}, $d) if $output; + } + +} + +open (my $fh, ">", "$output~") || die("can't open $output~ for writing"); + +export($fh, \@exports); + +close ($fh) || die("error writing"); +rename "$output~",$output || die("error overwriting $output"); + +exit 0; + +# ###################################################################### + +sub uniq { + my %seen; + return grep { !$seen{$_}++ } @_; +} + +sub export { + my $fp = shift; + my @exports = @{shift @_}; + + print $fp < +#include +END + foreach $h (uniq map { $_->{header} } @exports) { + print $fp "#include \"$h\"\n"; + } + + # this is effectively a function overloading system so that the + # compiler can select the datatype of the evaluation of the + # definition. + print $fp <%a, type=>'f32'", \\ + __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), double), "value=>%a, type=>'f64'", \\ + __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), int32_t), "value=>%d, type=>'i32'", \\ + __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), uint32_t), "value=>%d, type=>'u32'", \\ + __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), int64_t), "value=>%ld, type=>'i64'", \\ + __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), uint64_t), "value=>%ld, type=>'u64'", \\ + __builtin_choose_expr(__builtin_types_compatible_p(typeof(x), typeof(char[])), "value=>'%s', type=>'string'", "type => undefined" ))))))) + +int main(int argc, char **argv) { + FILE *fp = fopen(argv[1], "w"); + + fputs("{\\n", fp); +END + + foreach $export (@exports) { + print $fp <{name}' => { name => '$export->{name}', type => 'define', values => [\\n", fp); +END + foreach $d (@{$export->{export}}) { + my $docomment; + + if ($d->{comment}) { + my $comment = $d->{comment}; + $comment =~ s@(['"])@\\\\\\$1@g; + $comment =~ s@\n@\\n@g; + $docomment = ", comment => '$comment' "; + } + + print $fp < \\"$d->{name}\\", ", fp); + fprintf(fp, FMT($d->{name}), ($d->{name})); + fputs("$docomment},\\n", fp); +END + } + print $fp < 'flags' +# keep_comments => 1 to keep comments +# } +# returns a list of +# { +# name=>'name', +# comment='comment', +# file='filename', +# line=linenumber +# } +sub scanDefines { + my $header = shift; + my %o = %{shift @_}; + my $lastc = ""; + my $source; + my $sourceLine; -$all = join('|', @matchInclude); + print STDERR "Scanning $header\n"; -$matchInclude = qr/($all)/; + open (my $in,"-|","cpp -dD ".($o{keep_comments} ? '-CC' : '')." $o{CPPFLAGS} $header") || die("Can't find include file: $header"); + while (<$in>) { + # line markers + if (m/^\# (\d+) \"([^\"]*)/) { + $sourceLine = $1; + $source = $2; + next; + } + # accumulate comments + # single line comments override multi-line + if ($o{keep_comments}) { + if (m@/\*(.*)\*/@) { + do { + $lastc = $1 if $lastc eq ""; + s@/\*.*\*/@@; + } while (m@/\*(.*)\*/@); + } elsif (m@/\*(.*)@) { + my $com = "$1\n" if $1; + while (<$in>) { + chop; + if (m@(.*)\*/@) { + $com .= $1 if $1; + last; + } else { + $com .= "$_\n"; + } + } + $lastc = $com if $com && $lastc eq ""; + } elsif (m@//(.*)@) { + $lastc = $1 if $1; + s@//.*@@; + } + } -print "all: $all\n"; -print "$matchDefine\n"; + if (m/^\s*#define\s+(\w*)\(/) { + # ignore macros + $lastc = ""; + } elsif (m/^\s*#define\s+(\w*)\s*$/) { + # ignore empty defines + $lastc = ""; + } elsif (m/^\s*#define\s+(\w+)/) { + my $name = $1; + my %define = (); -sub readMatchFile { - my $path = shift @_; - my @lines = (); + $define{name} = $name; + $define{comment} = $lastc if $lastc ne ""; + $define{file} = $source; + $define{line} = $sourceLine; - open(my $f,"<$path"); - while (<$f>) { - chop; - next if m/^#/; + push @defines, \%define; - push @lines, $_; - } - close($f); + $lastc = ""; + } - my $all = join ('|', @lines); + $sourceLine++; + } + close $in; - return qr/^($all)$/; + return \@defines; } -$dump = 0; - -$args = join("", map { "\"-I$_\"" } @includes); - -open IN,"gcc -C -E -dD $args $header|"; -# only #defines that are defined in the same directory or file as header? -while () { - if (m/^# (\d*) \"(.*)\"/) { - $dump = ($2 =~ /$matchInclude/); - } - if ($dump) { - if (m/^#define\s+(\w*)\s+(.*)/) { - my $name = $1; - my $def = $2; - - if ($name =~ /$matchDefine/) { - if ($def =~ m/L$/) { - print "public static final long $name = (long)$def;\n"; if ($def =~ m/L$/); - } elsif ($def =~ m/^(0x)?[0-9]*$/) { - print "public static final long $name = (int)$def;\n"; if ($def =~ m/L$/); - } elsif ($def =~ m/f$/) { - print "public static final float $name = $def;\n"; if ($def =~ m/L$/); - } elsif ($def =~ m/^[\.E-+0-9]*$/) { +# TODO: library +sub loadControlFile { + my $path = shift @_; + my %def = (); + my $target; + + open (my $d,"<",$path); + + while (<$d>) { + next if /\s*\#/; + + chop; + + if ($target) { + if (m/\s*\}\s*$/) { + undef $target; + } elsif (/^\s*(\S+)\s*(.*)/) { + my @options = split(/\s+/,$2); + push @{$target->{items}}, { + match => $1, + options => \@options + }; + } + } elsif (/^(\w+)\s+(\S*)\s*(.*)\s+\{/) { + my @options = split(/\s+/,$3); + + $target = { + type => $1, + name => $2, + options => \@options, + items => [] + }; + push @{$def{$1}}, $target; + } elsif (/\S/) { + die("invalid line: %_"); } - #if ($def =~ m/^0x/) { - #} - } } - } + + close $d; + + return \%def; } -close IN; diff --git a/src/export.cc b/src/export.cc index 4a34c58..a37f0bd 100644 --- a/src/export.cc +++ b/src/export.cc @@ -697,7 +697,7 @@ static void export_type(tree type, const char *names) { names, names, size, TYPE_UNSIGNED(target) ? 'u' : 'i', size); for (tree v = TYPE_VALUES(target); v != NULL; v = TREE_CHAIN (v)) { - generate("\t{ label => '%s', value => '%ld' },\n", + generate("\t{ name => '%s', value => '%ld' },\n", IDENTIFIER_POINTER(TREE_PURPOSE(v)), tree_to_shwi(TREE_VALUE(v))); } @@ -873,7 +873,7 @@ static void export_type(tree type, const char *names) { names, names, size, TYPE_UNSIGNED(type) ? 'u' : 'i', size); for (tree v = TYPE_VALUES(type); v != NULL; v = TREE_CHAIN (v)) { - generate("\t{ label => '%s', value => '%ld' },\n", + generate("\t{ name => '%s', value => '%ld' },\n", IDENTIFIER_POINTER(TREE_PURPOSE(v)), tree_to_shwi(TREE_VALUE(v))); } @@ -977,7 +977,7 @@ static void plugin_finish(void *event_data, void *user_data) { for (struct node *n = dumped.list.head; n; n=n->next) generate("# %s\n", n->name); - generate(");\n"); + generate("}\n"); fclose(output_file); if (debug_level > 0) @@ -1007,7 +1007,7 @@ int plugin_init(struct plugin_name_args *plugin_info, struct plugin_gcc_version exit(EXIT_FAILURE); } - generate("%%data = (\n"); + generate("{\n"); register_callback(plugin_info->base_name, PLUGIN_FINISH_DECL, plugin_finish_decl, NULL); register_callback(plugin_info->base_name, PLUGIN_FINISH_TYPE, plugin_finish_type, NULL); diff --git a/src/generate-native b/src/generate-native index 55eaee7..62cf2fb 100755 --- a/src/generate-native +++ b/src/generate-native @@ -2,8 +2,14 @@ # -*- Mode:perl; perl-indent-level:4;tab-width:4; -*- -# TODO: get/set should take a typed field for struct/union * -# TODO: a flag for func_name to funcName option +# TODO: global defaults and/or pattern matching on api targets, partially done +# TODO: perhaps an option which just dumps everything it finds at some level +# TODO: the code here isn't a complete mess but could be improved +# - parameterise some functions +# TODO: map int to boolean where appropriate +# TODO: arrays with specified lengths passed as arguments could be size-checked in function stubs. +# TODO: error codes -> exceptions? +# TODO: auto-loading of libraries (library xx load=blah) option. use Data::Dumper; use File::Basename; @@ -14,11 +20,13 @@ $scriptPath = dirname(__FILE__); $package = ""; $output = "bin"; $verbose = 0; +$apidef = "api.def"; # usage # -t package target package # -d directory output root # -v verbose +# -a datafile add datafile to the dataset, can be from export.so or export-defines, etc my %typeSizes = ( i8 => 'byte', u8 => 'byte', @@ -29,6 +37,26 @@ my %typeSizes = ( f64 => 'double', ); +# or just use some formatting function table +my %defineType = ( + %typeSizes, + string => 'String' + ); + +my %definePrefix = ( + i8 => '(byte)', + u8 => '(byte)', + i16 => '(short)', + u16 => '(short)', + string => '"' + ); +my %defineSuffix = ( + u64 => 'L', + i64 => 'L', + f32 => 'f', + string => '"' + ); + my %intSizes = ( 8 => 'byte', 16 => 'short', 32 => 'int', 64 => 'long' ); my %typePrimitive = ( "byte" => 8, @@ -50,28 +78,337 @@ my %typeSignature = ( "MemoryAddress" => "Ljdk/incubator/foreign/MemoryAddress;", ); +my %renameTable = ( + 'studly-caps' => sub { my $s = shift; $s =~ s/(?:^|_)(.)/\U$1/g; return $s; }, + 'camel-case' => sub { my $s = shift; $s =~ s/(?:_)(.)/\U$1/g; return $s; }, + 'upper-leadin' => sub { my $s = shift; $s =~ s/^([^_]+)/\U$1/; return $s; }, + 'identity' => sub { return $_[0]; }, + ); + +my %data = (); + while (@ARGV) { - my $cmd = shift(@ARGV); + my $cmd = shift; if ($cmd eq "-t") { - $package = shift(@ARGV); + $package = shift; } elsif ($cmd eq "-d") { - $output = shift(@ARGV); + $output = shift; } elsif ($cmd eq "-v") { $verbose++; + } elsif ($cmd eq "-a") { + my $file = shift; + my $info; + + print "Add $file\n"; + + unless ($info = do $file) { + die "couldn't parse $file: $@" if $@; + die "couldn't import $file: $!" unless defined $info; + die "couldn't run $file" unless $info; + } + + %data = (%data, %{$info}); } else { - $meta = $cmd; + $apidef = $cmd; } } -# load in interface file -do $meta; +my $api = loadControlFile($apidef); +analyseAPI($api); analyseAndFixTypes(); -if ($verbose) { +if (0) { + $s = $data{'struct:AVCodecContext'}; + $obj = findAPIObject($api, 'struct', $s->{name}); + print Dumper($obj); + + foreach $m (@{$s->{fields}}) { + $inc = findAPIField($obj, $m->{name}); + print " $m->{name} - $inc\n"; + } +} + +my $toDump = analyseDependencies(\%data, findRoots($api)); + +if ($verbose > 1) { print "Using:n"; print Dumper(\%data); + print "API:n"; + print Dumper($api); +} + +# find api.struct that matches a given struct name +sub findAPIObject { + my $api = shift; + my $type = shift; + my $name = shift; + + print "find api for $type:$name\n" if $verbose; + foreach $obj ( @{$api->{$type}} ) { + next if $obj->{name} eq ''; + print " $obj->{name} ? $name\n" if $verbose; + if ($obj->{name} =~ m@/(.*)/@) { + my $rx = qr/$1/; + return $obj if ($name =~ m/$rx/); + } elsif ($obj->{name} eq $name) { + return $obj; + } + } + + print " -> fallback=$type:\n" if $verbose && $api->{"$type:"}; + return $api->{"$type:"}; +} + +# sub findAPIStruct { +# my $api = shift; +# my $name = shift; + +# foreach $obj ( @{$api->{struct}} ) { +# next if $obj->{name} eq ''; +# print "$obj->{name} ? $name\n" if $verbose; +# if ($obj->{name} =~ m@/(.*)/@) { +# my $rx = qr/$1/; +# return $obj if ($name =~ m/$rx/); +# } elsif ($obj->{name} eq $name) { +# return $obj; +# } +# } + +# return $api->{'struct:'}; +# } + +sub findAPIField { + my $obj = shift; + my $name = shift; + + foreach $inc (grep { $_->{mode} eq 'field' } @{$obj->{items}}) { + return $inc if $name =~ m/$inc->{regex}/; + } +} + +# find all directly referenced types and field types from api +sub findRoots { + my $api = shift; + my %seen; + + foreach $obj ( @{$api->{library}} ) { + foreach $inc ( @{$obj->{items}} ) { + if ($inc->{mode} eq 'func') { + my @list = grep { $_->{type} eq $inc->{mode} && $_->{name} =~ m/$inc->{regex}/ } values %data; + + foreach $func (@list) { + $seen{"func:$func->{name}"} ++; + } + } + } + } + + # all defines included + foreach $def ( @{$api->{define}} ) { + $seen{"define:$def->{name}"} ++; + } + + foreach $obj ( @{$api->{struct}}, @{$api->{func}}) { + my @list; + + if ($obj->{name} =~ m@/(.*)/@) { + my $rx = "$obj->{type}:$1"; + push @list, grep { $_ =~ m/$rx/ } keys %data; + } else { + push @list, "$obj->{type}:$obj->{name}"; + } + + foreach $n (@list) { + $seen{$n} ++; + } + + } + + delete $seen{'struct:'}; + delete $seen{'func:'}; + delete $seen{'call:'}; + + my @list = sort keys %seen; + + return \@list; +} + +# analyse dependencies of the supplied roots +# only fields actually referenced in the api.def file are included +# \%seen = \%data, \@roots +sub analyseDependencies { + my $data = shift; + my @roots = @{shift @_}; + my %seen; + + print "Finding dependencies of $#roots roots\n"; + + while ($#roots >= 0) { + my $name = shift @roots; + my $s = $data{$name}; + my @list; + + next if $seen{$name}++; + + print "visit $name $s->{name}\n" if $verbose; + + if ($s->{type} =~ m/struct|union/) { + my $obj = findAPIObject($api, 'struct', $s->{name}); + if ($obj->{default} eq 'all') { + push @list, @{$s->{fields}}; + } else { + push @list, grep { findAPIField($obj, $_->{name}) } @{$s->{fields}}; + } + } elsif ($s->{type} =~ m/func|call/) { + @list = @{$s->{arguments}}; + push @list, $s->{result}; + } + + foreach $m (@list) { + my $type = $m->{type}; + + print " item $m->{name} '$type'\n" if $verbose; + if ($m->{ctype} =~ m/enum (.*)/) { + $type = "enum:$1"; + } + + push @roots,$type if $data{$type}; + } + } + + foreach $name (sort grep { m/:/ } keys %seen) { + print " $name\n"; + } + print "\n"; + return \%seen; +} + +# find which api->thing->items applies to a given field name, if any +sub findAPIItem { + my $api = shift; + my $type = shift; + my $target = shift; + my $mode = shift; + my $name = shift; + + #print "search for $target.$name in $type.$mode\n"; + # what about exclude? + foreach $obj ( @{$api->{$type}} ) { + if ($obj->{name} eq $target) { + #print " found $target\n"; + foreach $inc (grep { $_->{mode} eq $mode } @{$obj->{items}}) { + #print " check $inc->{match}\n"; + return $inc if $name =~ m/$inc->{regex}/; + } + } + } +} + +sub analyseAPI { + my $api = shift; + + foreach $obj ( @{$api->{struct}}, @{$api->{library}}, @{$api->{func}}) { + $obj->{access} = 'rw'; + $obj->{default} = 'all'; + $obj->{rename} = $renameTable{'identity'}; + $obj->{'func:rename'} = $renameTable{'identity'}; + $obj->{'field:rename'} = $renameTable{'identity'}; + foreach $o (@{$obj->{options}}) { + if ($o =~ m/^default=(none|all)$/) { + $obj->{default} = $1; + } elsif ($o =~ m/^access=([rwi]+)$/) { + $obj->{access} = $1; + } elsif ($o =~ m@^(rename|field:rename|func:rename)=(.*)@) { + my $target = $1; + + if ($obj->{name} eq 'SwsContext') { + print "SwsContext rename = $o\n"; + } + + foreach $n (split /,/,$2) { + my $old = $obj->{$target}; + my $new = $renameTable{$n}; + + if ($n =~ m@^s/(.*)/(.*)/$@) { + my $rx = qr/$1/; + my $rp = $2; + $obj->{$target} = sub { my $s=shift; $s = $old->($s); $s =~ s/$rx/$rp/; return $s;}; + } elsif ($new) { + $obj->{$target} = sub { my $s=shift; $s = $old->($s); return $new->($s); }; + } + } + } + } + + my $defmode = $obj->{type} eq 'library' ? 'func' : 'field'; + + foreach $inc (@{$obj->{items}}) { + if ($inc->{match} =~ m@^(field|func|define|struct|enum):/(.*)/$@) { + $inc->{regex} = qr/$2/; + $inc->{mode} = $1; # ?? "$1-include"; + } elsif ($inc->{match} =~ m@^(field|func|define|struct|enum):(.*)$@) { + $inc->{regex} = qr/^$2$/; + $inc->{mode} = $1; + } elsif ($inc->{match} =~ m@^/(.*)/$@) { + $inc->{regex} = qr/$1/; + $inc->{mode} = $defmode; + } else { + $inc->{regex} = qr/^$inc->{match}$/; + $inc->{mode} = $defmode; + } + + $inc->{rename} = $renameTable{'identity'}; + $inc->{scope} = 'static' if $obj->{type} eq 'library'; + + # maybe depends on mode above + foreach $o (@{$inc->{options}}) { + if ($o =~ m/^access=([rwi])+/) { + $inc->{access} = $1; + } elsif ($o =~ m/^rename=(.*)/) { + foreach $n (split /,/,$1) { + my $old = $inc->{rename}; + my $new = $renameTable{$n}; + + if ($n =~ m@^s/(.*)/(.*)/$@) { + my $rx = qr/$1/; + my $rp = $2; + $inc->{rename} = sub { my $s=shift; $s = $old->($s); $s =~ s/$rx/$rp/; return $s;}; + } elsif ($new) { + $inc->{rename} = sub { my $s=shift; $s = $old->($s); return $new->($s); }; + } else { + my $x = $n; + $inc->{rename} = sub { return $x; }; + } + } + } elsif ($o =~ m/^array-size=(.*)/) { + $inc->{'array_size'} = $1; + } elsif ($o =~ m/^array$/) { + $inc->{'array'} = 1; + } elsif ($o =~ m/^instance=(.*)/) { + $inc->{instance} = $1; + } elsif ($o =~ m/^static$/) { + $inc->{scope} = 'static'; + } elsif ($o =~ m/^constructor=(.*)$/) { + $inc->{constructor} = $1; + } elsif ($o =~ m/^constructor-result=(.*)$/) { + $inc->{constructor_result} = $1; + } elsif ($o =~ m/^success=(.*)$/) { + $inc->{success} = $1; + } + # exclude mode, etc + } + + $inc->{rename} = $obj->{"$inc->{mode}:rename"} if $inc->{rename} == $renameTable{'identity'} && $obj->{"$inc->{mode}:rename"}; + } + + if ($obj->{name} eq '') { + $api->{"$obj->{type}:"} = $obj; + } + } + + $api->{'call:'} = { rename => $renameTable{'identity'}, scope => 'static'} if !$api->{'call:'}; } # anonymous structs @@ -104,7 +441,6 @@ sub analyseAndFixTypes { my $s = $data{$n}; my @list; - if ($s->{type} =~ m/struct|union/) { @list = @{$s->{fields}}; } elsif ($s->{type} =~ m/func|call/) { @@ -127,7 +463,7 @@ sub analyseAndFixTypes { } # must be last - $a->{typeInfo} = queryTypeInfo($a); + $a->{typeInfo} = analyseTypeInfo($s, $a); } } @@ -158,14 +494,18 @@ sub formatTypeLayout { if ($m->{deref} =~ m/^(u64|u32):/) { $desc .= "Memory.POINTER$withName"; } elsif ($m->{type} =~ m/^([iuf]\d+)$/) { - if ($m->{deref} =~ m/\[(\d*).*\]/) { + if ($m->{deref} =~ m/\[(\d*)u64:.*\]/) { + $desc .= "MemoryLayout.sequenceLayout($1, Memory.POINTER)$withName"; + } elsif ($m->{deref} =~ m/\[(\d*).*\]/) { $desc .= "MemoryLayout.sequenceLayout($1, Memory.".uc($typeSizes{$m->{type}}).")$withName"; } else { $desc .= 'Memory.'.uc($typeSizes{$m->{type}})."$withName"; } } elsif ($m->{type} =~ m/^(struct|union):(.*)/) { my $type = $2; - if ($m->{deref} =~ m/\[(\d*).*\]/) { + if ($m->{deref} =~ m/\[(\d*)u64:.*\]/) { + $desc .= "MemoryLayout.sequenceLayout($1, Memory.POINTER)$withName"; + } elsif ($m->{deref} =~ m/\[(\d*).*\]/) { $desc .= "MemoryLayout.sequenceLayout($1, $type.LAYOUT)$withName"; } else { $desc .= "$type.LAYOUT$withName"; @@ -222,9 +562,19 @@ sub formatSignature { return $desc; } -sub queryTypeInfo { - my $m = shift @_; +# TODO: perhaps ByteArray should just be MemorySegment, kinda painful to wrap them all the time +sub analyseTypeInfo { + my $s = shift; + my $m = shift; my $info = {}; + my $inc; + + #print " query $s->{name} $s->{type} '$m->{name}', '$m->{type}'\n"; + if ($s->{type} eq 'struct') { + $inc = findAPIItem($api, 'struct', $s->{name}, 'field', $m->{name}); + } elsif ($s->{type} eq 'func') { + $inc = findAPIItem($api, 'func', $s->{name}, 'field', $m->{name}); + } # default for everything not specifically handled $info->{carrier} = "MemoryAddress"; @@ -234,21 +584,43 @@ sub queryTypeInfo { # This is a function pointer, type must be type = 'call:.*' if ($m->{type} =~ m/^call:(.*)/) { $info->{type} = "Memory.FunctionPointer<$1>"; - $info->{create} = "$1.downcall(\${result}, scope())"; + $info->{create} = "$1.downcall(\${result}, \${scope})"; } else { die(); } } elsif ($m->{type} =~ m/^([iuf]\d+)$/) { - if ($m->{deref} =~ m/\[(\d*).*\]/) { + if ($m->{deref} =~ m/\[(\d*)u64:.*\]/) { + $info->{byValue} = 1; + $info->{type} = "Memory.PointerArray"; + $info->{create} = $info->{type}.".create(\${result})"; + } elsif ($m->{deref} =~ m/\[(\d*).*\]/) { + # TODO: some mode thing rather than byvalue? $info->{byValue} = 1; $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array"; $info->{create} = $info->{type}.".create(\${result})"; } elsif ($m->{deref} =~ m/^(u64:u64:|u32:u32:)/) { $info->{type} = "Memory.PointerArray"; - $info->{create} = $info->{type}.".create(\${result})"; + $info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})"; } elsif ($m->{deref} =~ m/^(u64:|u32:)/) { - $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array"; - $info->{create} = $info->{type}.".create(\${result})"; + # assume any char * is a string unless an array-size or array is specified + if ($inc->{array}) { + $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array"; + $info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})"; + } elsif ($inc->{array_size}) { + $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array"; + $info->{create} = $info->{type}.".createArray(\${result}, \${array_size}, \${scope})"; + } elsif ($typeSizes{$m->{type}} eq 'byte') { + $info->{type} = 'String'; + $info->{resolve} = "(Addressable)Memory.address(frame.copy(\${value}))"; + $info->{create} = "(\${result}).getUtf8String(0)"; + $info->{resolveFrame} = 1; + # for a function or a constructor that uses this element + $s->{resolveFrame} = 1; + } else { + # ideally length 0 but panama-foreign doesn't grok that so fuckit + $info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array"; + $info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})"; + } } else { $info->{type} = $typeSizes{$m->{type}}; $info->{carrier} = $typeSizes{$m->{type}}; @@ -257,23 +629,42 @@ sub queryTypeInfo { } } elsif ($m->{type} =~ m/^(struct|union):(.*)/) { my $type = $2; - if ($m->{deref} =~ m/\[(\d*).*\]/) { - $info->{type} = $type; - $info->{create} = $info->{type}.".createArray($1, \${result}, scope())"; + if ($m->{deref} =~ m/\[(\d*)(.*)\]/) { + $info->{byValue} = 1; + # handle 'type name[x]' and 'type *name[x]' + my $count = $1; + my $deref = $2; + if ($deref =~ m/^u64:u64:/) { + die("can't handle double-deref array"); + } elsif ($deref =~ m/^u64:/) { + $info->{type} = "Memory.HandleArray<$type>"; + $info->{create} = "Memory.HandleArray.create(\${result}, $type\:\:create, \${scope})"; + } else { + $info->{type} = $type; + #$info->{create} = $info->{type}.".create(\${result}, \${scope})"; + $info->{create} = $info->{type}.".create(\${result})"; + } } elsif ($m->{deref} =~ m/^(u64:u64:|u32:u32:)/) { - $info->{type} = "Memory.PointerArray"; - $info->{create} = $info->{type}.".create(\${result})"; + # this assumes ** is as far as it gets + $info->{type} = "Memory.HandleArray<$type>"; + if ($inc->{array_size}) { + $info->{create} = "Memory.HandleArray.createArray(\${result}, \${array_size}, $type\:\:create, \${scope})"; + } else { + $info->{create} = "Memory.HandleArray.createArray(\${result}, Long.MAX_VALUE, $type\:\:create, \${scope})"; + } } elsif ($m->{deref} =~ m/^(u64:|u32:)/) { $info->{type} = $type; - $info->{create} = $info->{type}.".create(\${result}, scope())"; + $info->{create} = $info->{type}.".create(\${result}, \${scope})"; } else { + # FIXME: change this to a reftype or something + $info->{byValue} = 1; $info->{type} = $type; $info->{create} = $info->{type}.".create(\${result})"; } } elsif ($m->{type} eq "void") { if ($m->{deref} =~ m/^(u64:u64:|u32:u32:)/) { $info->{type} = "Memory.PointerArray"; - $info->{create} = $info->{type}.".create(\${result})"; + $info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})"; } elsif ($m->{deref} =~ m/^(u64:|u32:)/) { $info->{type} = "MemoryAddress"; $info->{create} = "\${result}"; @@ -291,45 +682,160 @@ sub queryTypeInfo { return $info; } -sub formatFunction { - my $c = shift @_; +# TODO: see if this can be merged with formatFunction +# TODO: should constructor take a resourcescope always? +# TODO: exceptions for errors +sub formatConstructor { + my $c = shift; + my $inc = shift; my @arguments = @{$c->{arguments}}; my $result = $c->{result}; my $desc; my $index = 0; - my $desc; + my $count = 0; + my $desc = ""; my $name = $c->{name}; - my $rtype = $result->{typeInfo}->{type}; + my $otype = $data{$arguments[$inc->{constructor_result}]->{type}}->{name}; - #print Dumper($c); - $desc = $rtype; + $name = $inc->{rename}->($name); + + $desc .= "static " if $inc->{scope} eq 'static'; + $desc .= $otype; $desc .= " $name("; for $m (@arguments) { - $desc .= ", " if ($index++ > 0); - $desc .= $m->{typeInfo}->{type}; - $desc .= " $m->{name}" + if (($inc->{constructor} && $index != $inc->{constructor_result})) { + $desc .= ", " if ($count++ > 0); + $desc .= $m->{typeInfo}->{type}; + $desc .= " $m->{name}" + } + $index++; + } + if ($inc->{constructor}) { + $desc .= ", " if ($count++ > 0); + $desc .= "ResourceScope scope"; } $desc .=") {\n "; + + $desc .= "$result->{typeInfo}->{carrier} res\$value;\n" if ($rtype ne "void"); + + $desc .= " try "; + $desc .= "(Frame frame = Memory.createFrame()) " if ($c->{resolveFrame}); + $desc .= "{\n"; + $desc .= " Memory.HandleArray<$otype> res\$holder = Memory.HandleArray.createArray(1, frame, $otype\:\:create, scope);\n" if ($inc->{constructor}); + $desc .= " res\$value = ($result->{typeInfo}->{carrier})" if ($rtype ne "void"); + $desc .= " " if ($rtype eq "void"); + $index = 0; + $desc .= "$c->{name}\$FH.invokeExact(\n "; + for $m (@arguments) { + my $resolve = $m->{typeInfo}->{resolve}; + + if ($inc->{constructor} && $index == $inc->{instance}) { + $desc .= ",\n " if ($index++ > 0); + $desc .= "(Addressable)res\$holder.address()"; + } elsif ($inc->{scope} ne 'static' && $index == $inc->{instance}) { + $desc .= ",\n " if ($index++ > 0); + $desc .= "(Addressable)address()"; + } else { + $desc .= ",\n " if ($index++ > 0); + + if ($resolve) { + $resolve =~ s/\$\{value\}/$m->{name}/g; + $desc .= $resolve; + } else { + $desc .= "$m->{name}"; + } + } + } + $desc .= ");\n"; + + if ($rtype ne "void" && defined $inc->{success}) { + # my $create = $result->{typeInfo}->{create}; + + # # ooh, templates could insert other arguments or values as well? + # $create =~ s/\$\{result\}/res\$value/; + # if ($inc->{scope} eq 'static') { + # $create =~ s/\$\{scope\}/ResourceScope.globalScope()/; + # } else { + # $create =~ s/\$\{scope\}/scope()/; + # } + + foreach $code (split /,/,$inc->{success}) { + $desc .= " if (res\$value == $code) return res\$holder.getAtIndex(0);\n"; + } + } else { + $desc .= " return res\$holder.getAtIndex(0);\n"; + } + # throw Error()? + $desc .= " } catch (Throwable t) { throw new RuntimeException(t); }\n"; + + # throw failures here based on res$value + $desc .= " return null;\n"; + + $desc .="}"; + + #print "$desc\n"; + return $desc; +} + +sub formatFunction { + my $c = shift; + my $inc = shift; + my @arguments = @{$c->{arguments}}; + my $result = $c->{result}; + my $desc; + my $index = 0; + my $count = 0; + my $desc = ""; + my $name = $c->{name}; + my $rtype = $result->{typeInfo}->{type}; + + if ($inc->{constructor}) { + return formatConstructor($c, $inc); + } + + $name = $inc->{rename}->($name); + + $desc .= "static " if $inc->{scope} eq 'static'; + $desc .= $rtype; + $desc .= " $name("; + + for $m (@arguments) { + if ($inc->{scope} eq 'static' || $index != $inc->{instance}) { + $desc .= ", " if ($count++ > 0); + $desc .= $m->{typeInfo}->{type}; + $desc .= " $m->{name}" + } + $index++; + } + $desc .=") {\n "; - $desc .= "try {\n"; + $desc .= "try "; + $desc .= "(Frame frame = Memory.createFrame()) " if ($c->{resolveFrame}); + $desc .= "{\n"; $desc .= " $result->{typeInfo}->{carrier} res\$value = ($result->{typeInfo}->{carrier})" if ($rtype ne "void"); $desc .= " " if ($rtype eq "void"); - $desc .= "$name\$FH.invokeExact(\n "; + $index = 0; + $desc .= "$c->{name}\$FH.invokeExact(\n "; for $m (@arguments) { my $resolve = $m->{typeInfo}->{resolve}; - $desc .= ",\n " if ($index++ > 0); - - if ($resolve) { - $resolve =~ s/\$\{value\}/$m->{name}/g; - $desc .= $resolve; + if ($inc->{scope} ne 'static' && $index == $inc->{instance}) { + $desc .= ",\n " if ($index++ > 0); + $desc .= "(Addressable)address()"; } else { - $desc .= "$m->{name}" + $desc .= ",\n " if ($index++ > 0); + + if ($resolve) { + $resolve =~ s/\$\{value\}/$m->{name}/g; + $desc .= $resolve; + } else { + $desc .= "$m->{name}"; + } } } $desc .= ");\n"; @@ -338,7 +844,12 @@ sub formatFunction { my $create = $result->{typeInfo}->{create}; # ooh, templates could insert other arguments or values as well? - $create =~ s/\${result\}/res\$value/; + $create =~ s/\$\{result\}/res\$value/; + if ($inc->{scope} eq 'static') { + $create =~ s/\$\{scope\}/ResourceScope.globalScope()/; + } else { + $create =~ s/\$\{scope\}/scope()/; + } $desc .= " return $create;\n"; } @@ -351,9 +862,10 @@ sub formatFunction { } # create an interface for function pointers -# FiXME: this should be exportCallback to a file +# FiXME: this should be exportCallback to a file? sub formatCallback { - my $c = shift @_; + my $c = shift; + my $obj = shift; my @arguments = @{$c->{arguments}}; my $result = $c->{result}; my $desc; @@ -421,6 +933,7 @@ sub formatCallback { my $create = $m->{typeInfo}->{create}; $create =~ s/\$\{result\}/$m->{name}/g; + $create =~ s/\$\{scope\}/scope/g; $desc .= ",\n " if ($index++ > 0); $desc .= "$create"; @@ -450,12 +963,13 @@ sub formatCallback { $desc .= " symbol,\n"; # HACK: this is basically the same as any function call, just patch in the changes for now - $tmp = formatFunction($c); + $tmp = formatFunction($c, $obj); $tmp =~ s/^(.*) ($name)\(/(/; $tmp =~ s/\) \{/) -> {/; $tmp =~ s/^/ /mg; $desc .= $tmp; + $desc .= "\n"; $desc .= " );\n"; $desc .= " }\n"; @@ -591,72 +1105,91 @@ sub formatLayout { } sub formatGetSet { - my $s = shift @_; - my $m = shift @_; + my $s = shift; + my $m = shift; + my $rename = shift; + my $access = shift; + my $inc = shift; my $desc = ""; my $info = $m->{typeInfo}; - my $Name = ucfirst($m->{name}); + my $Name = ucfirst($rename); my $tmp; # info -> needsalloc? - # TODO: String # TODO: embedded arrays are quite different setup if ($info->{byValue}) { $tmp = $info->{create}; $tmp =~ s/\$\{result\}/segment/g; + $tmp =~ s/\$\{scope\}/scope()/g; $desc .= " public $info->{type} get$Name() {\n"; - $desc .= " MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n"; - $desc .= " MemorySegment segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n"; + #$desc .= " MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n"; + #$desc .= " MemorySegment segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n"; + $desc .= " MemorySegment segment = this.segment.asSlice($m->{name}\$byteOffset, $m->{name}\$byteSize);\n"; $desc .= " return $tmp;\n"; $desc .= " }\n"; - $desc .= " public $info->{type} get$Name"."At(long index) {\n"; - $desc .= " MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n"; - $desc .= " MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n"; - $desc .= " segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n"; - $desc .= " return $tmp;\n"; - $desc .= " }\n"; - } else { + if ($access =~ m/i/) { + $desc .= " public $info->{type} get$Name"."At(long index) {\n"; + #$desc .= " MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n"; + #$desc .= " MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n"; + #$desc .= " segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n"; + $desc .= " MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index + $m->{name}\$byteOffset, $m->{name}\$byteSize);\n"; + $desc .= " return $tmp;\n"; + $desc .= " }\n"; + } + } elsif ($access =~ /r/) { $tmp = $info->{create}; + $tmp =~ s/\$\{result\}/($info->{carrier})$m->{name}\$VH.get(segment)/g; + $tmp =~ s/\$\{scope\}/scope()/g; + # fixme: lookup type of array size? somewhere? doesn't matter i think + $tmp =~ s/\${array_size}/(long)$inc->{array_size}\$VH.get(segment)/g; $desc .= " public $info->{type} get$Name() {\n"; $desc .= " return $tmp;\n"; $desc .= " }\n"; - $desc .= " public $info->{type} get$Name"."At(long index) {\n"; - $desc .= " MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n"; - $desc .= " return $tmp;\n"; - $desc .= " }\n"; + if ($access =~ m/i/) { + $desc .= " public $info->{type} get$Name"."At(long index) {\n"; + $desc .= " MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n"; + $desc .= " return $tmp;\n"; + $desc .= " }\n"; + } } - if (!($m->{deref} =~ m/\[(\d*).*\]/)) { + if ($access =~ m/w/ && !$info->{byValue}) { $tmp = $info->{resolve}; $tmp =~ s/\$\{value\}/value/g; $desc .= " public void set$Name($info->{type} value) {\n"; + $desc .= " try (Frame frame = Memory.createFrame()) {\n" if ($info->{resolveFrame}); $desc .= " $m->{name}\$VH.set(segment, $tmp);\n"; + $desc .= " }\n" if ($info->{resolveFrame}); $desc .= " }\n"; - $desc .= " public void set$Name"."At(long index, $info->{type} value) {\n"; - $desc .= " MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n"; - $desc .= " $m->{name}\$VH.set(segment, $tmp);\n"; - $desc .= " }\n"; + if ($access =~ m/i/) { + $desc .= " public void set$Name"."At(long index, $info->{type} value) {\n"; + $desc .= " try (Frame frame = Memory.createFrame()) {\n" if ($info->{resolveFrame}); + $desc .= " MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n"; + $desc .= " $m->{name}\$VH.set(segment, $tmp);\n"; + $desc .= " }\n" if ($info->{resolveFrame}); + $desc .= " }\n"; + } } - # indexed - return $desc; } sub exportStruct { - my $f = shift @_; - my $s = shift @_; + my $f = shift; + my $s = shift; + my $obj = shift; my @fields = @{$s->{fields}}; my $isHandle = $s->{size} == 0; + my $doArray = $obj->{access} =~ m/i/; #my @functions = @{shift @_}; print $f "package $package;\n" if $package; @@ -673,11 +1206,13 @@ sub exportStruct { print $f " private $s->{name}(MemorySegment segment) { this.segment = segment; }\n"; print $f " public static $s->{name} create(MemorySegment segment) { return new $s->{name}(segment); }\n"; print $f " public static $s->{name} create(MemoryAddress address, ResourceScope scope) {\n"; - print $f " return create(MemorySegment.ofAddress(address, LAYOUT.byteSize(), scope));\n"; - print $f " }\n"; - print $f " public static $s->{name} createArray(MemoryAddress address, long size, ResourceScope scope) {\n"; - print $f " return create(MemorySegment.ofAddress(address, size * LAYOUT.byteSize(), scope));\n"; + print $f " return MemoryAddress.NULL != address ? create(MemorySegment.ofAddress(address, LAYOUT.byteSize(), scope)) : null;\n"; print $f " }\n"; + if ($doArray) { + print $f " public static $s->{name} createArray(MemoryAddress address, long size, ResourceScope scope) {\n"; + print $f " return MemoryAddress.NULL != address ? create(MemorySegment.ofAddress(address, size * LAYOUT.byteSize(), scope)) : null;\n"; + print $f " }\n"; + } print $f " public static $s->{name} create(Frame frame) { return create(frame.allocate(LAYOUT)); }\n"; print $f " public static $s->{name} create(ResourceScope scope) { return create(MemorySegment.allocateNative(LAYOUT, scope)); }\n"; print $f " public MemoryAddress address() { return segment.address(); }\n"; @@ -687,91 +1222,81 @@ sub exportStruct { print $f " MemoryAddress address;\n"; print $f " ResourceScope scope;\n"; # constructors - print $f " private $s->{name}(MemoryAddress address, ResourceScope scope) { this.address = address; this.scope = scope}\n"; - print $f " public static $s->{name} create(MemoryAddress address) { return new $s->{name}(address); }\n"; + print $f " private $s->{name}(MemoryAddress address, ResourceScope scope) { this.address = address; this.scope = scope;}\n"; + print $f " public static $s->{name} create(MemoryAddress address, ResourceScope scope) { return MemoryAddress.NULL != address ? new $s->{name}(address, scope) : null; }\n"; print $f " public MemoryAddress address() { return address; }\n"; print $f " public ResourceScope scope() { return scope; }\n"; } - # FIXME: use typeInfo - # TODO: indexed accessors - # accessors - if (1) { - foreach $m (@fields) { - print $f formatGetSet($s, $m); - } - } else { - foreach $m (@fields) { - my $Name = ucfirst($m->{name}); + my %seen; - print $f " // [$m->{deref}] [$m->{type}] [$m->{ctype}]\n"; + # Any defines + foreach $inc (grep { $_->{mode} eq 'define' } @{$obj->{items}}) { + my $def = $data{$inc->{match}}; - if ($m->{deref} =~ m/^(u64:|u32:)\(/) { - # This is a function pointer, type must be type = 'call:.*' + die ("unknown define $inc->{match} in $s->{name}\n") if !$def; - if ($m->{type} =~ m/^call:(.*)/) { - my $jtype = $1; + delete $toDump->{$inc->{match}}; - $jtype =~ s/(.*)\((.*)\)(.*)/Call$1_$2_$3/; + foreach $m (@{$def->{values}}) { + print $f " /**\n ($m->{comment}) */\n" if ($m->{comment}); + print $f " public static final $defineType{$m->{type}} $m->{name} = $definePrefix{$m->{type}}$m->{value}$defineSuffix{$m->{type}};\n"; + } + } - print $f " public Memory.FunctionPointer<$jtype> get$Name() {\n"; - print $f " // FIXME: better scope\n"; - print $f " return $jtype.downcall((MemoryAddress)$m->{name}\$VH.get(segment), scope());\n"; - print $f " }\n"; + # TODO: any enums we want to include here I suppose? - print $f " public void set$Name(Memory.FunctionPointer<$jtype> value) {\n"; - print $f " $m->{name}\$VH.set(segment, Memory.address(value));\n"; - print $f " }\n"; - } - } elsif ($m->{deref} =~ m/^(u64:|u32:)/) { - # all other pointer types require extra context, e.g. a length or type - print $f " public MemoryAddress get$Name() {\n"; - print $f " return (MemoryAddress)$m->{name}\$VH.get(segment);\n"; - print $f " }\n"; - - # FIXME: set could use the type though - print $f " public void set$Name(MemoryAddress value) {\n"; - print $f " $m->{name}\$VH.set(segment, Memory.address(value));\n"; - print $f " }\n"; - } elsif ($m->{type} eq "bitfield") { - # TODO - } elsif ($m->{type} =~ m/^struct|union:(.*)$/) { - my $jtype = $1; - - $jtype = "Memory.HandleArray<$jtype>" if ($data{$m->{type}}->{size} == 0); - - # embedded type including arrays - print $f " public $jtype get$Name() {\n"; - print $f " MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n"; - print $f " MemorySegment seg = segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n"; - print $f " return $jtype.create(seg, $jtype::new);\n" if ($data{$m->{type}}->{size} == 0); - print $f " return $jtype.create(seg);\n" if ($data{$m->{type}}->{size} != 0); - print $f " }\n"; - } elsif ($m->{type} =~ m/^[uif]\d+$/) { - my $jtype = $typeSizes{$m->{type}}; - my $Jtype = ucfirst($jtype); - my $JTYPE = uc($jtype); - - if ($m->{deref} =~ m/\[(\d*).*\]/) { - # array type - print $f " public Memory.$Jtype"."Array get$Name() {\n"; - print $f " MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n"; - print $f " MemorySegment seg = segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n"; - print $f " return new Memory.$Jtype"."Array(seg);\n"; - print $f " }\n"; - } else { - # primitive type - print $f " public $jtype get$Name() {\n"; - print $f " return ($jtype)$m->{name}\$VH.get(segment);\n"; - print $f " }\n"; - - print $f " public void set$Name($jtype value) {\n"; - print $f " $m->{name}\$VH.set(segment, ($jtype)value);\n"; - print $f " }\n"; - } + # Accessors + foreach $m (@fields) { + my $access = $obj->{access}; + my $rename = $obj->{'field:rename'}; + my $matches = 0; + my $matchinc; + + # check for match + foreach $inc (grep { $_->{mode} eq 'field' } @{$obj->{items}}) { + $matches = $m->{name} =~ m/$inc->{regex}/; + + if ($matches) { + $access = $inc->{access} if $inc->{access}; + $rename = $inc->{rename} if $inc->{rename} != $renameTable{'identity'}; + $matchinc = $inc; + last; } - # struct print $f " MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n"; + } + + my $output = $matches || ($obj->{default} eq 'all'); + + if ($output) { + my $name = $rename ? $rename->($m->{name}) : $m->{name}; + print $f formatGetSet($s, $m, $name, $access, $matchinc); + } + } + + # Functions + foreach $inc (grep { $_->{mode} eq 'func' } @{$obj->{items}}) { + my @list; + + print "$obj->{name} match $inc->{match} regex $inc->{regex}\n" if $verbose; + + if ($data{$inc->{match}}) { + push @list, $data{$inc->{match}}; + } else { + @list = grep { $_->{name} =~ m/$inc->{regex}/ } values %data; + } + + foreach $c (@list) { + my $tmp; + + next if $seen{$c->{name}}++; + + print $f " static final MethodHandle $c->{name}\$FH = Memory.downcall(\"$c->{name}\",\n"; + $tmp = formatFunctionDescriptor($c); + print $f "$tmp);\n"; + + $tmp = formatFunction($c, $inc); + print $f 'public '.$tmp."\n\n"; } } @@ -780,11 +1305,59 @@ sub exportStruct { print $f "static final GroupLayout LAYOUT = ".formatLayout($s).";\n"; foreach $m (@fields) { - next if ($m->{typeInfo}->{byValue}); - print $f " static final VarHandle $m->{name}\$VH = LAYOUT.varHandle(MemoryLayout.PathElement.groupElement(\"$m->{name}\"));\n"; + print $f " // type='$m->{type}' deref='$m->{deref}' info->type ='$m->{typeInfo}->{type}'\n"; + if ($m->{typeInfo}->{byValue}) { + print $f " static final long $m->{name}\$byteOffset = " + ." LAYOUT.byteOffset(MemoryLayout.PathElement.groupElement(\"$m->{name}\"));\n"; + print $f " static final long $m->{name}\$byteSize = " + ."LAYOUT.select(MemoryLayout.PathElement.groupElement(\"$m->{name}\")).byteSize();\n"; + } else { + print $f " static final VarHandle $m->{name}\$VH = " + ."LAYOUT.varHandle(MemoryLayout.PathElement.groupElement(\"$m->{name}\"));\n"; + } } } + # verification? + if (!$isHandle) { + print $f <{name}\", $m->{offset});\n"; + } +my $bytes = $s->{size}/8; + print $f <{name}.sizeof = %d != $bytes", LAYOUT.byteSize()), null); + } +END + } + print $f "}\n"; +} + +# file,enum +sub exportEnum { + my $f = shift; + my $s = shift; + my @values = @{$s->{values}}; + my $jtype = $typeSizes{$s->{value_type}}; + my $prefix = $definePrefix{$s->{value_type}}; + my $suffix = $definePrefix{$s->{value_type}}; + print $f "package $package;\n" if $package; + + print $f "public interface $s->{name} {\n"; + + foreach $v (@values) { + print $f " public static final $jtype $v->{name} = $prefix$v->{value}$suffix;\n"; + } + print $f "}\n"; } @@ -825,22 +1398,46 @@ sub nameToPath { return $name; } -foreach $x (grep { m/^(struct|union):/ } sort keys %data) { - my $s = $data{$x}; - my $path = nameToPath($output, "$package.$s->{name}"); +#print "api\n"; +#print Dumper($api); - open (my $f, ">", $path) || die ("Cannot open '$path' for writing"); +# Dump struct type +foreach $obj ( @{$api->{struct}} ) { + my @list; - exportStruct($f, $s); + next if $obj->{name} eq ''; - close $f; -} + if ($obj->{name} =~ m@/(.*)/@) { + my $rx = qr/struct:$1/; -foreach $x (grep { m/^call:/ } sort keys %data) { - my $c = $data{$x}; - my $name = $c->{name}; + @list = map { s/struct://; $_ } grep { $_ =~ m/$rx/ } keys %data; + } else { + push @list, $obj->{name}; + } - my $path = nameToPath($output, "$package.$name"); + foreach $name (@list) { + my $path = nameToPath($output, "$package.$name"); + my $s = $data{"struct:$name"}; + + #print Dumper($obj); + + delete $toDump->{"struct:$name"}; + + if ($s) { + open (my $f, ">", $path) || die ("Cannot open '$path' for writing"); + + exportStruct($f, $s, $obj); + + close $f; + } else { + print "No struct $name\n"; + } + } +} + +# Dump library type +foreach $lib ( @{$api->{library}} ) { + my $path = nameToPath($output, "$package.$lib->{name}"); open (my $f, ">", $path) || die ("Cannot open '$path' for writing"); @@ -848,22 +1445,57 @@ foreach $x (grep { m/^call:/ } sort keys %data) { print $f "import jdk.incubator.foreign.*;\n"; print $f "import java.lang.invoke.*;\n"; - print $f formatCallback($c); + print $f "public class $lib->{name} {\n"; + + print $f " static ResourceScope scope() { return ResourceScope.globalScope(); }\n"; + + # scan for matches + foreach $inc (@{$lib->{items}}) { + if ($inc->{mode} eq 'func') { + my @list = grep { $_->{type} eq $inc->{mode} && $_->{name} =~ m/$inc->{regex}/ } values %data; + foreach $c (@list) { + my $tmp; + + print $f " static final MethodHandle $c->{name}\$FH = Memory.downcall(\"$c->{name}\",\n"; + $tmp = formatFunctionDescriptor($c); + print $f $tmp.");\n"; + + $tmp = formatFunction($c, $inc); + print $f "public "; + print $f $tmp."\n\n"; + } + } elsif ($inc->{mode} eq 'define') { + my @list = grep { $_->{type} eq $inc->{mode} && $_->{name} =~ m/$inc->{regex}/ } values %data; + foreach $c (@list) { + delete $toDump->{"define:$c->{name}"}; + foreach $m (@{$c->{fields}}) { + print $f " /**\n ($m->{comment}) */\n" if ($m->{comment}); + print $f " public static final $defineType{$m->{type}} $m->{name} = $definePrefix{$m->{type}}$m->{value}$defineSuffix{$m->{type}};\n"; + } + } + } + } + + print $f "}\n"; close $f; } -# just quick and dirty for now -# may want a non-static version with a specific scope? -{ - my @functions = grep { /^func:/ } keys %data; - my $lib = { - name => "APILib", - functions => \@functions - }; +print "remaining dependent types\n"; +foreach $k (sort grep { !m/func:/ } keys %{$toDump}) { + print " $k\n"; +} +print "\n"; +# Calls referenced +foreach $k (sort keys %{$toDump}) { + next if (!($k =~ m/call:(.+)/)); - my $path = nameToPath($output, "$package.$lib->{name}"); + my $name = $1; + my $c = $data{$k}; + my $obj = findAPIObject($api, 'call', $name); + + my $path = nameToPath($output, "$package.$name"); open (my $f, ">", $path) || die ("Cannot open '$path' for writing"); @@ -871,23 +1503,117 @@ foreach $x (grep { m/^call:/ } sort keys %data) { print $f "import jdk.incubator.foreign.*;\n"; print $f "import java.lang.invoke.*;\n"; - print $f "public class $lib->{name} {\n"; + print $f formatCallback($c, $obj); - print $f " static ResourceScope scope() { return ResourceScope.globalScope(); }\n"; - foreach $cname (@{$lib->{functions}}) { - my $c = $data{$cname}; - my $tmp; + close $f; +} + +# any struct remaining in toDump (but not in api) +# FIXME: how to lookup obj? +foreach $k (sort keys %{$toDump}) { + if ($k =~ m/struct:(.*)/) { + my $name = $1; + my $s = $data{$k}; + my $path = nameToPath($output, "$package.$name"); + + die("missing struct $name") if !$s; - print $f " static final MethodHandle $c->{name}\$FH = Memory.downcall(\"$c->{name}\",\n"; - $tmp = formatFunctionDescriptor($c); - print $f "$tmp);\n"; + delete $toDump->{$k}; - $tmp = formatFunction($c); - print $f "public static "; - print $f $tmp; + my $obj = findAPIObject($api, 'struct', $name); + + open (my $f, ">", $path) || die ("Cannot open '$path' for writing"); + exportStruct($f, $s, $obj); + close $f; } +} - print $f "}\n"; +# Dump enum types used by everything and not dumped elsehwere +foreach $k (sort keys %{$toDump}) { + if ($k =~ m/enum:(.*)/) { + my $name = $1; + my $s = $data{$k}; + my $path = nameToPath($output, "$package.$name"); - close $f; + die("missing enum $name") if !$s; + + open(my $f, ">", $path) || die ("Cannot open '$path' for writing"); + + exportEnum($f, $s); + + close $f; + } +} + +# Dump define types not dumped elsehwere +foreach $k (sort keys %{$toDump}) { + if ($k =~ m/define:(.*)/) { + my $name = $1; + my $s = $data{$k}; + my $path = nameToPath($output, "$package.$name"); + + die("missing define $name") if !$s; + + open(my $f, ">", $path) || die ("Cannot open '$path' for writing"); + + print $f "package $package;\n" if $package; + print $f "public interface $s->{name} {\n"; + + foreach $m (@{$s->{values}}) { + # some parsing problem here + next if !$m->{value}; + + print $f " /**\n ($m->{comment}) */\n" if ($m->{comment}); + print $f " public static final $defineType{$m->{type}} $m->{name} = $definePrefix{$m->{type}}$m->{value}$defineSuffix{$m->{type}};\n"; + } + + print $f "}\n"; + + close $f; + } +} + +# and we're done +exit 0; + +sub loadControlFile { + my $path = shift @_; + my %def = (); + my $target; + + open (my $d,"<",$path); + + while (<$d>) { + next if /\s*\#/; + + chop; + + if ($target) { + if (m/\s*\}\s*$/) { + undef $target; + } elsif (/^\s*(\S+)\s*(.*)/) { + my @options = split(/\s+/,$2); + push @{$target->{items}}, { + match => $1, + options => \@options + }; + } + } elsif (/^(\w+)\s+(\S*)\s*(.*)\s+\{/) { + my @options = split(/\s+/,$3); + + $target = { + type => $1, + name => $2, + options => \@options, + items => [] + }; + push @{$def{$1}}, $target; + } elsif (/\S/) { + die("invalid line: %_"); + } + } + + close $d; + + return \%def; } diff --git a/src/template/Memory.java b/src/template/Memory.java index 1cdf9f6..023c828 100644 --- a/src/template/Memory.java +++ b/src/template/Memory.java @@ -181,7 +181,7 @@ public class Memory { this.segment = segment; } - public static ByteArray create(MemorySegment segment) { + public static ByteArray create(MemorySegment segment) { return new ByteArray(segment); } @@ -242,16 +242,24 @@ public class Memory { public static class ShortArray extends AbstractList implements Memory.Addressable { final MemorySegment segment; - public ShortArray(MemorySegment segment) { + private ShortArray(MemorySegment segment) { this.segment = segment; } - public ShortArray(Frame frame, long size) { - this(frame.allocateArray(Memory.SHORT, size)); + public static ShortArray create(MemorySegment segment) { + return new ShortArray(segment); } - public ShortArray(Frame frame, short... values) { - this(frame.allocateArray(Memory.SHORT, values)); + public static ShortArray createArray(MemoryAddress address, long length, ResourceScope scope) { + return create(MemorySegment.ofAddress(address, length, scope)); + } + + public static ShortArray createArray(long length, SegmentAllocator alloc) { + return create(alloc.allocateArray(Memory.SHORT, length)); + } + + public static ShortArray create(SegmentAllocator alloc, short... values) { + return create(alloc.allocateArray(Memory.SHORT, values)); } public final MemoryAddress address() { @@ -295,16 +303,24 @@ public class Memory { public static class IntArray extends AbstractList implements Memory.Addressable { final MemorySegment segment; - public IntArray(MemorySegment segment) { + private IntArray(MemorySegment segment) { this.segment = segment; } - public IntArray(Frame frame, long size) { - this(frame.allocateArray(Memory.INT, size)); + public static IntArray create(MemorySegment segment) { + return new IntArray(segment); + } + + public static IntArray createArray(MemoryAddress address, long length, ResourceScope scope) { + return create(MemorySegment.ofAddress(address, length, scope)); + } + + public static IntArray createArray(long length, SegmentAllocator alloc) { + return create(alloc.allocateArray(Memory.INT, length)); } - public IntArray(Frame frame, int... values) { - this(frame.allocateArray(Memory.INT, values)); + public static IntArray create(SegmentAllocator alloc, int... values) { + return create(alloc.allocateArray(Memory.INT, values)); } public final MemoryAddress address() { @@ -352,12 +368,20 @@ public class Memory { this.segment = segment; } - public LongArray(Frame frame, long size) { - this(frame.allocateArray(Memory.LONG, size)); + public static LongArray create(MemorySegment segment) { + return new LongArray(segment); } - public LongArray(Frame frame, long... values) { - this(frame.allocateArray(Memory.LONG, values)); + public static LongArray createArray(MemoryAddress address, long length, ResourceScope scope) { + return create(MemorySegment.ofAddress(address, length, scope)); + } + + public static LongArray createArray(long length, SegmentAllocator alloc) { + return create(alloc.allocateArray(Memory.LONG, length)); + } + + public static LongArray create(SegmentAllocator alloc, long... values) { + return create(alloc.allocateArray(Memory.LONG, values)); } public final MemoryAddress address() { @@ -401,16 +425,24 @@ public class Memory { public static class FloatArray extends AbstractList implements Memory.Addressable { final MemorySegment segment; - public FloatArray(MemorySegment segment) { + private FloatArray(MemorySegment segment) { this.segment = segment; } - public FloatArray(Frame frame, long size) { - this(frame.allocateArray(Memory.FLOAT, size)); + public static FloatArray create(MemorySegment segment) { + return new FloatArray(segment); + } + + public static FloatArray createArray(MemoryAddress address, long length, ResourceScope scope) { + return create(MemorySegment.ofAddress(address, length, scope)); + } + + public static FloatArray createArray(long length, SegmentAllocator alloc) { + return create(alloc.allocateArray(Memory.FLOAT, length)); } - public FloatArray(Frame frame, float... values) { - this(frame.allocateArray(Memory.FLOAT, values)); + public static FloatArray create(SegmentAllocator alloc, float... values) { + return create(alloc.allocateArray(Memory.FLOAT, values)); } public final MemoryAddress address() { @@ -454,16 +486,24 @@ public class Memory { public static class DoubleArray extends AbstractList implements Memory.Addressable { final MemorySegment segment; - public DoubleArray(MemorySegment segment) { + private DoubleArray(MemorySegment segment) { this.segment = segment; } - public DoubleArray(Frame frame, long size) { - this(frame.allocateArray(Memory.DOUBLE, size)); + public static DoubleArray create(MemorySegment segment) { + return new DoubleArray(segment); + } + + public static DoubleArray createArray(MemoryAddress address, long length, ResourceScope scope) { + return create(MemorySegment.ofAddress(address, length, scope)); } - public DoubleArray(Frame frame, double... values) { - this(frame.allocateArray(Memory.DOUBLE, values)); + public static DoubleArray createArray(long length, SegmentAllocator alloc) { + return create(alloc.allocateArray(Memory.DOUBLE, length)); + } + + public static DoubleArray create(SegmentAllocator alloc, double... values) { + return create(alloc.allocateArray(Memory.DOUBLE, values)); } public final MemoryAddress address() { @@ -515,8 +555,16 @@ public class Memory { return new PointerArray(segment); } - public static PointerArray createArray(long size, SegmentAllocator alloc) { - return create(alloc.allocateArray(Memory.POINTER, size)); + public static PointerArray createArray(MemoryAddress address, long length, ResourceScope scope) { + return create(MemorySegment.ofAddress(address, length, scope)); + } + + public static PointerArray createArray(long length, SegmentAllocator alloc) { + return create(alloc.allocateArray(Memory.POINTER, length)); + } + + public static PointerArray create(Frame alloc, MemoryAddress... values) { + return create(alloc.allocateArray(Memory.POINTER, values)); } public final MemoryAddress address() { @@ -557,23 +605,38 @@ public class Memory { } } + // This needs a separate scope from the array itself public static class HandleArray extends AbstractList implements Memory.Addressable { final MemorySegment segment; + final ResourceScope scope; BiFunction create; - private HandleArray(MemorySegment segment, BiFunction create) { + private HandleArray(MemorySegment segment, BiFunction create, ResourceScope scope) { this.segment = segment; this.create = create; + this.scope = scope; } public static HandleArray create(MemorySegment segment, BiFunction create) { - return new HandleArray<>(segment, create); + return new HandleArray<>(segment, create, segment.scope()); + } + + public static HandleArray create(MemorySegment segment, BiFunction create, ResourceScope scope) { + return new HandleArray<>(segment, create, scope); } public static HandleArray createArray(long size, SegmentAllocator alloc, BiFunction create) { return create(alloc.allocateArray(Memory.POINTER, size), create); } + public static HandleArray createArray(long size, SegmentAllocator alloc, BiFunction create, ResourceScope scope) { + return create(alloc.allocateArray(Memory.POINTER, size), create, scope); + } + + public static HandleArray createArray(MemoryAddress address, long size, BiFunction create, ResourceScope scope) { + return create(MemorySegment.ofAddress(address, size * Memory.POINTER.byteSize(), scope), create); + } + @Override public final MemoryAddress address() { return segment.address(); diff --git a/test-api/Makefile b/test-api/Makefile index a458279..a7a8666 100644 --- a/test-api/Makefile +++ b/test-api/Makefile @@ -13,17 +13,23 @@ api_demo_SOURCES := $(wildcard src/api/test/*.java) all:: mkdir -p bin -all:: bin/demo.built +all:: bin/demo.built bin/libapi.so bin/api.built: bin/api.gen $(JAVAC) $(JAVACFLAGS) -cp bin/classes -d bin/classes \ $(shell find bin/java -name '*.java') touch $@ -bin/api.gen: bin/api.pm ../src/generate-native $(api_SOURCES) - ../src/generate-native -d bin/java -t proto.api ./bin/api.pm +bin/api.gen: bin/api.pm ../src/generate-native $(api_SOURCES) bin/api-defines.pm api.api + ../src/generate-native -v -d bin/java -t proto.api -a ./bin/api.pm -a ./bin/api-defines.pm api.api touch $@ +bin/api-defines.pm: api.h ../src/export-defines api.api + ../src/export-defines -d bin/api-defines.c api.api + $(CC) -o bin/api-defines -I. bin/api-defines.c + bin/api-defines $@~ + mv $@~ $@ + bin/api.pm: api.h ../src/export.so gcc -fplugin=../src/export.so -fplugin-arg-export-output=$@~ ./$< -o /dev/null mv $@~ $@ diff --git a/test-api/README b/test-api/README new file mode 100644 index 0000000..59a3438 --- /dev/null +++ b/test-api/README @@ -0,0 +1,8 @@ + +Introduction +------------ + +Small generator and prototyping area for a c library. + +This is mostly to test out ideas like pattern matching for mostly +automatic api conversion. diff --git a/test-api/api.api b/test-api/api.api new file mode 100644 index 0000000..cfe3103 --- /dev/null +++ b/test-api/api.api @@ -0,0 +1,15 @@ + +# collect all functions and the defines we want +library APILib load=api { + define:api-defines + func:// +} + +# grab all structures +struct // default=all field:rename=studly-caps { +} + +# grab all defines in api.h +define api-defines api.h { + api.h file-include +} diff --git a/test-api/api.c b/test-api/api.c index 0aad6b8..58cac5f 100644 --- a/test-api/api.c +++ b/test-api/api.c @@ -1,6 +1,7 @@ #include #include +#include #include "api.h" static void funca(int a) { diff --git a/test-api/api.h b/test-api/api.h index e4639ca..4e6b023 100644 --- a/test-api/api.h +++ b/test-api/api.h @@ -1,4 +1,13 @@ +#define API_FOO "12" +#define API_MAKEVERSION(a, b) (a <<16) | b +#define API_VERSION API_MAKEVERSION(1, 0) +#define API_A 12UL +#define API_B 12L +#define API_C 12U +#define API_D 12.0 +#define API_E 12.0f + struct data { struct data *next; @@ -22,3 +31,9 @@ struct api { void *api_func(const char *name); struct api *api_create(void); + +struct list_node { + struct list_node *succ; + struct list_node *pred; + char name[64-16]; +}; diff --git a/test-api/src/api/test/TestAPI.java b/test-api/src/api/test/TestAPI.java index 2d8be3b..108d898 100644 --- a/test-api/src/api/test/TestAPI.java +++ b/test-api/src/api/test/TestAPI.java @@ -35,7 +35,7 @@ public class TestAPI { b.setC((byte)255); b.setD((byte)255); - b.setTest_a(cb); + b.setTestA(cb); System.out.println("from a"); print_data(a); @@ -49,7 +49,7 @@ public class TestAPI { // dynamic lookup System.out.println("call funca via symbol lookup"); - Memory.FunctionPointer funca = Call_i32_v.downcall(api_func(Memory.ByteArray.create("funca", frame)), scope); + Memory.FunctionPointer funca = Call_i32_v.downcall(api_func("funca"), scope); System.out.printf(" %s\n", funca.symbol()); funca.function().call(12); diff --git a/test-ffmpeg/Makefile b/test-ffmpeg/Makefile new file mode 100644 index 0000000..e1403fc --- /dev/null +++ b/test-ffmpeg/Makefile @@ -0,0 +1,52 @@ + +CFLAGS=-g -fPIC +HOST_CC=gcc + +JAVA_HOME=/opt/jdk-foreign/jvm/openjdk-19-internal +JAVAC=$(JAVA_HOME)/bin/javac +JAVA=$(JAVA_HOME)/bin/java + +JAVACFLAGS=--add-modules jdk.incubator.foreign + +api_SOURCES := $(wildcard ../src/api/*.java) +ffmpeg_SOURCES := $(wildcard src/proto/ffmpeg/*.java) +ffmpeg_demo_SOURCES := $(wildcard src/ffmpeg/test/*.java) + +all:: + mkdir -p bin + +all:: bin/demo.built + +bin/ffmpeg.built: bin/ffmpeg.gen $(ffmpeg_SOURCES) + $(JAVAC) $(JAVACFLAGS) -cp bin/classes -d bin/classes \ + $(shell find bin/java -name '*.java') \ + $(ffmpeg_SOURCES) + touch $@ + +bin/ffmpeg.gen: bin/ffmpeg.pm bin/ffmpeg-defines.pm ../src/generate-native $(api_SOURCES) + ../src/generate-native -d bin/java -t proto.ffmpeg -a ./bin/ffmpeg.pm -a ./bin/ffmpeg-defines.pm ffmpeg.api + touch $@ + +bin/ffmpeg-defines.pm: ffmpeg.h ../src/export-defines ffmpeg.api + ../src/export-defines -d bin/ffmpeg-defines.c ffmpeg.api + $(HOST_CC) -o bin/ffmpeg-defines -I. bin/ffmpeg-defines.c + bin/ffmpeg-defines $@~ + mv $@~ $@ + +bin/ffmpeg.pm: ffmpeg.h ../src/export.so + gcc -fplugin=../src/export.so -fplugin-arg-export-output=$@~ ./$< -o /dev/null + mv $@~ $@ + +bin/demo.built: $(ffmpeg_demo_SOURCES) bin/ffmpeg.built + $(JAVAC) $(JAVACFLAGS) -cp bin/classes -d bin/classes $(ffmpeg_demo_SOURCES) + touch $@ + +demo: all + $(JAVA) --enable-native-access=ALL-UNNAMED --add-modules jdk.incubator.foreign \ + -cp bin/classes \ + ffmpeg.test.TestFFMPEG + +clean: + rm -rf bin 000[0-9].pnm + +.PHONY: demo clean all diff --git a/test-ffmpeg/README b/test-ffmpeg/README new file mode 100644 index 0000000..e22691f --- /dev/null +++ b/test-ffmpeg/README @@ -0,0 +1,12 @@ + +Introduction +------------ + +Small demonstrator of generate-native on a complex and messy api +wrapped in a Java "object oriented" one. + +make demo +--------- + +Copy some video file to 'movie.avi' to run the demo. It will output +the first 10 frames to "\d{5}.pnm". diff --git a/test-ffmpeg/ffmpeg.api b/test-ffmpeg/ffmpeg.api new file mode 100644 index 0000000..777d287 --- /dev/null +++ b/test-ffmpeg/ffmpeg.api @@ -0,0 +1,222 @@ + +# access=rwi +# r read (get) +# w write (set) +# i read/write indexed (get/setAtIndex) + +struct field:rename=studly-caps func:rename=camel-case access=rw { +} + +struct AVFormatContext default=none func:rename=s/^(avformat_|av_)//,camel-case field:rename=studly-caps { + iformat access=r rename=InputFormat + oformat access=r rename=OutputFormat + pb rename=IOContext + + ctx_flags access=r + nb_streams access=r rename=NumStreams + + streams array-size=nb_streams + + start_time access=r + duration access=r + bit_rate access=r + + interrupt_callback + + func:avformat_open_input static constructor=byref constructor-result=0 success=0 + func:avformat_find_stream_info rename=findStreamInfo instance=0 + + func:av_read_frame +} + +struct AVStream default=none field:rename=studly-caps { + index access=r + id access=r rename=ID + time_base + start_time + duration + nb_frames rename=NumFrames + discard + avg_frame_rate rename=AverageFrameRate + sample_aspect_ratio + codecpar rename=CodecParameters +} + +struct AVCodec access=r default=none func:rename=s/^avcodec_//,camel-case field:rename=studly-caps { + id rename=ID + name + long_name + type + capabilities + max_lowres + # need some sort of length=null-terminated here i suppose + supported_framerates rename=framerates + pix_fmts rename=pixelFormats + + func:av_codec_next static rename=next + func:avcodec_find_decoder static + func:avcodec_find_decoder_by_name static + # here or on AVCodecContext? + func:avcodec_alloc_context3 rename=allocContext + + define:AVCodecBits +} + +struct AVCodecContext default=none func:rename=s/^avcodec_//,camel-case field:rename=studly-caps { + codec_id rename=CodecID + + skip_loop_filter + skip_idct + skip_frame + + func:avcodec_alloc_context3 static rename=alloc + func:avcodec_open2 rename=open + func:avcodec_send_packet + func:avcodec_receive_packet + func:avcodec_send_frame + func:avcodec_receive_frame + + define:AVCodecContextBits +} + +struct AVFrame default=all func:rename=s/^av_frame_//,camel-case field:rename=studly-caps { + func:av_frame_alloc static + func:av_frame_free +} + +struct AVFrameSideData default=none { +} + +struct AVRational field:rename=studly-caps { +} + +struct AVCodecParameters func:rename=s/^avcodec_parameters_//,camel-case field:rename=studly-caps { + codec_id rename=CodecID + func:avcodec_parameters_alloc static + func:avcodec_parameters_free + func:avcodec_parameters_copy + func:avcodec_parameters_to_context instance=1 + func:avcodec_parameters_from_context +} + +struct AVPacket default=all func:rename=s/^av_packet_//,camel-case field:rename=studly-caps { + pts|dts rename=upper-leadin + data array-size=size + + func:av_packet_alloc static + func:av_packet_free + func:av_init_packet rename=init +} + +struct SwsContext func:rename=s/^sws_// { +# func:/^sws_/ + func:sws_getContext static + func:sws_freeContext + func:sws_scale + + define:sws +} + +struct SwsFilter { +} +struct SwsVector { +} +struct AVPixFmtDescriptor { +} +struct AVComponentDescriptor access=rwi { +} + +library AVUtil { + func:/^av_image_/ +} + +struct AVProbeData default=none field:rename=studly-caps { +} + +struct AVBufferRef access=rwi default=none { +} +struct AVPacketSideData default=none { +} +struct AVIOContext default=none { +} +struct AVIOInterruptCB default=none { +} + +struct AVDictionaryEntry { +} + +struct AVDictionary func:rename=s/^av_dict_//,camel-case { + define:dict + func:/av_dict_/ +} + +struct AVInputFormat default=none access=r field:rename=studly-caps { + name + long_name + mime_type + flags + extensions +# want to define functions here + func:av_iformat_next static rename=next + func:av_register_input_format instance=0 rename=register +} + +struct AVOutputFormat default=none field:rename=studly-caps { + name + long_name + mime_type + flags + extensions + audio_codec + video_codec + subtitle_codec +} + +define dict ffmpeg.h { + /AV_DICT_/ +} + +define AVChannelLayoutBits ffmpeg.h { + /^AV_CH_LAYOUT_/ x64 +} + +define AVErrorBits ffmpeg.h { + /^AVERROR_/ x32 +} + +define AVCodecContextBits ffmpeg.h { + /^AV_CODEC_FLAG_|AV_CODEC_FLAG2_/ u32 + /^AV_INPUT_BUFFER_/ i32 +} + +define AVCodecBits ffmpeg.h { + /^AV_CODEC_CAP_/ x32 +} + +define AVIOContextBits ffmpeg.h { + /^AVSEEK_|AVIO_FLAG_|AVIO_SEEKABLE_/ +} + +define AVOptionsBits ffmpeg.h { + /^AV_OPT_/ +} + +define sws ffmpeg.h { + /^SWS_/ +} + +# This is used to define 'char *' arguments which are not strings. +# ... a bit clumsy + +func av_image_copy_plane { + src|dst array +} +func av_image_copy_to_buffer { + dst array +} +func av_image_fill_arrays { + src array +} +func av_image_fill_pointers { + ptr array +} diff --git a/test-ffmpeg/ffmpeg.h b/test-ffmpeg/ffmpeg.h new file mode 100644 index 0000000..1d3a58e --- /dev/null +++ b/test-ffmpeg/ffmpeg.h @@ -0,0 +1,6 @@ + +#include +#include +#include +#include +#include diff --git a/test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java b/test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java new file mode 100644 index 0000000..cbec649 --- /dev/null +++ b/test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java @@ -0,0 +1,137 @@ + +package ffmpeg.test; + +import jdk.incubator.foreign.*; +import java.io.FileOutputStream; +import java.io.IOException; + +import proto.ffmpeg.Memory.*; +import proto.ffmpeg.*; + +import static proto.ffmpeg.AVMediaType.*; +import static proto.ffmpeg.AVPixelFormat.*; + +public class TestFFMPEG { + + public static void main(String[] args) throws Exception { + System.loadLibrary("avcodec"); + System.loadLibrary("avformat"); + System.loadLibrary("swscale"); + System.loadLibrary("avutil"); + + try (Frame frame = Memory.createFrame(); + ResourceScope scope = ResourceScope.newConfinedScope()) { + int res; + + if (false) { + AVInputFormat avif = null; + while ((avif = AVInputFormat.next(avif)) != null) { + System.out.printf("name: %30s %s\n", + avif.getName(), + avif.getLongName()); + } + } + + AVFormatContext format = AVFormatContext.openInput("movie.avi", null, null, scope); + + if (format == null) { + throw new IOException("File not found: movie.avi"); + } + + format.findStreamInfo(null); + + HandleArray streams = format.getStreams(); + + System.out.printf(" streams=%d %d\n", format.getNumStreams(), streams.size()); + + for (AVStream s: streams) { + AVCodecParameters cp = s.getCodecParameters(); + + System.out.printf(" stream [index=%d, id=%d]\n", s.getIndex(), s.getID()); + System.out.printf(" start=%d\n duration=%d\n", s.getStartTime(), s.getDuration()); + + System.out.printf(" codecType: %d\n", cp.getCodecType()); + System.out.printf(" codecId: %d\n", cp.getCodecID()); + System.out.printf(" bitrate: %d\n", cp.getBitRate()); + System.out.printf(" samplerate: %d\n", cp.getSampleRate()); + } + + AVCodecParameters vcp = null; + AVStream vstream = null; + + for (AVStream stream: streams) { + AVCodecParameters cp = stream.getCodecParameters(); + + switch (cp.getCodecType()) { + case AVMEDIA_TYPE_VIDEO: + System.out.printf(" video = %d\n", stream.getIndex()); + vstream = stream; + vcp = cp; + break; + case AVMEDIA_TYPE_AUDIO: + System.out.printf(" audio = %d\n", stream.getIndex()); + break; + } + } + + AVCodec codec = AVCodec.findDecoder(vcp.getCodecID()); + //AVCodecContext c = AVCodecContext.alloc(codec); + AVCodecContext c = codec.allocContext(); + AVFrame avframe = AVFrame.alloc(); + AVPacket packet = AVPacket.alloc(); + + packet.init(); + vcp.toContext(c); + c.setCodecID(codec.getID()); + res = c.open(codec, null); + + System.out.printf("c.open = %d\n", res); + + AVPixelReader pix = null; + long nframes = 0; + long pts = 0; + + while ((res = format.readFrame(packet)) >= 0) { + int index = packet.getStreamIndex(); + + if (index == vstream.getIndex()) { + //boolean ready; + int ready; + + pts = packet.getDTS(); + + res = c.sendPacket(packet); + res = c.receiveFrame(avframe); + + if (res == 0) { + int w = avframe.getWidth(); + int h = avframe.getHeight(); + int fmt = avframe.getFormat(); + + if (true) { + if (pix == null) { + //pix = frame.getPixelReader(w, h, 0); + pix = new FramePixelReader(avframe, w, h, 0); + } + + byte[] pixels = new byte[w*h*3]; + + pix.getPixels(0, h, AV_PIX_FMT_RGB24, pixels, 0, 0); + + try (FileOutputStream fos = new FileOutputStream(String.format("%04d.pnm", nframes))) { + String header = String.format("P6\n%d %d\n255\n", w, h); + fos.write(header.getBytes()); + fos.write(pixels); + } + } + + System.out.printf("%06d video frame %dx%d @ %d\n", nframes, w, h, fmt); + nframes++; + } + } + if (nframes == 10) + break; + } + } + } +} diff --git a/test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java b/test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java new file mode 100644 index 0000000..6ca4a3c --- /dev/null +++ b/test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java @@ -0,0 +1,53 @@ +/* + * Copyright (C) 2017 Michael Zucchi + * + * This file is part of jjmpeg + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ +package proto.ffmpeg; + +import java.nio.ByteBuffer; + +/** + * AVPixelReader is a high level interface to SwsContext. + *

+ * This interface allows one to retrieve pixels by bands to a specific format. + * + * @see AVFrame#getPixelReader + */ +public interface AVPixelReader { + + /** + * Read pixels into a Buffer. The buffer must be a Direct*Buffer. + * + * @param + * @param y Start scanline. Currently only 0 works. + * @param h End scanline output. Currently only the output height works. + * @param fmt Pixel format to write to. + * @param buffer Target for pixels. + * @param scanlineStride Should match the output width and may be ignored. + */ + public void getPixels(int y, int h, int fmt, ByteBuffer buffer, int scanlineStride); + + public void getPixels(int y, int h, int fmt, byte[] buffer, int offset, int scanlineStride); + + public void getPixels(int y, int h, int fmt, short[] buffer, int offset, int scanlineStride); + + public void getPixels(int y, int h, int fmt, int[] buffer, int offset, int scanlineStride); + + public long bufferSize(int h, int fmt); + + public void release(); +} diff --git a/test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java b/test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java new file mode 100644 index 0000000..26ec177 --- /dev/null +++ b/test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java @@ -0,0 +1,153 @@ +/* + * Copyright (C) 2020 Michael Zucchi + * + * This file is part of jjmpeg + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ +package proto.ffmpeg; + +import java.nio.Buffer; +import java.nio.ByteBuffer; +import java.nio.BufferOverflowException; +import jdk.incubator.foreign.*; +import proto.ffmpeg.Memory.*; + +public class FramePixelReader implements AVPixelReader { + + SwsContext sws; + final AVFrame frame; + final int dwidth, dheight, flags; + int dfmt = -1; + + ResourceScope scope = ResourceScope.newImplicitScope(); + MemorySegment seg; + + public FramePixelReader(AVFrame frame, int dwidth, int dheight, int flags) { + this.frame = frame; + this.dwidth = dwidth; + this.dheight = dheight; + this.flags = flags; + } + + SwsContext getContext(int fmt) { + if (sws == null || fmt != dfmt) { + dfmt = fmt; + if (sws != null) { + //sws.release(); + } + sws = SwsContext.getContext(frame.getWidth(), frame.getHeight(), frame.getFormat(), + dwidth, dheight, dfmt, flags, + null, null, null); + } + return sws; + } + + MemorySegment getSegment(long size) { + if (seg == null || seg.byteSize() < size) { + if (seg != null) { + //Memory.free(seg.baseAddress()); + } + //seg = Memory.ofNative(Memory.malloc(size), size); + seg = MemorySegment.allocateNative(size, 16, scope); + } + return seg; + } + + private void getPixels(int y, int h, int fmt, MemorySegment buffer, int scanlineStride) { + SwsContext ctx = getContext(fmt); + + // FIXME: check sizes + + try (Frame a = Memory.createFrame()) { + IntArray stride = IntArray.createArray(4, a); + PointerArray data = PointerArray.createArray(4, a); + ByteArray buf = ByteArray.create(buffer); + int res; + + res = AVUtil.av_image_fill_linesizes(stride, fmt, dwidth); + res = AVUtil.av_image_fill_pointers(data, fmt, h, buf, stride); + + System.out.printf(" base %016x\n", buffer.address().toRawLongValue()); + for (int i=0;i<4;i++) { + System.out.printf(" %d %6d %016x\n", i, stride.get(i), data.get(i).address().toRawLongValue()); + } + + if (res > 0 && res <= buffer.byteSize()) { + ctx.scale( + frame.getData(), + frame.getLinesize(), + y, h, + data, stride); + } else { + throw new BufferOverflowException(); + } + } + } + + public long bufferSize(int h, int fmt) { + try (Frame a = Memory.createFrame()) { + IntArray stride = IntArray.createArray(4, a); + PointerArray data = PointerArray.createArray(4, a); + int res; + + res = AVUtil.av_image_fill_linesizes(stride, fmt, dwidth); + res = AVUtil.av_image_fill_pointers(data, fmt, h, null, stride); + + return res; + } + } + + public void getPixels(int y, int h, int fmt, ByteBuffer buffer, int scanlineStride) { + getPixels(y, h, fmt, MemorySegment.ofByteBuffer((ByteBuffer)buffer), scanlineStride); + } + + public void getPixels(int y, int h, int fmt, byte[] buffer, int offset, int scanlineStride) { + SwsContext ctx = getContext(fmt); + + try (Frame a = Memory.createFrame()) { + IntArray stride = IntArray.createArray(4, a); + PointerArray data = PointerArray.createArray(4, a); + MemorySegment seg = getSegment(bufferSize(h, fmt)); + ByteArray dst = ByteArray.create(seg); + int res; + + res = AVUtil.av_image_fill_linesizes(stride, fmt, dwidth); + res = AVUtil.av_image_fill_pointers(data, fmt, h, dst, stride); + + AVUtil.av_image_fill_linesizes(stride, fmt, dwidth); + + ctx.scale( + frame.getData(), + frame.getLinesize(), + y, h, + data, stride); + + MemorySegment.ofArray(buffer).copyFrom(seg); + } + } + + public void getPixels(int y, int h, int fmt, short[] buffer, int offset, int scanlineStride) { + } + + public void getPixels(int y, int h, int fmt, int[] buffer, int offset, int scanlineStride) { + } + + public void release() { + if (scope != null) { + scope.close(); + scope = null; + } + } +} -- 2.39.5