Planet CDOT

November 22, 2019


Brett Dennis

Telescope week 4

This week, working on the telescope project I have taken the time to look at a few more PR’s that have been in need of review. This one, this one, as well as this third one. I ahve also started to look for another task to tackle for the project internally, and have asked around to see if there is more that I can do in regards to the internal testing, and have checked with this issue to see if the maintainer is still working on it as it is a week old and I would like to attempt it.

In other news, I have started my search for another project to work on for the final release of 0.4. Nothing completely solid that I would want to work on for the moment, but I will find something.

The original issue that I had worked on for 0.2 and later 0.3 finally got closed for it’s intended purpose a little bit ago. Even if I wasn’t the one to completely finish off the issue, I was still thanked for the contributions I had made and the PR’s I had done to finish off another part of that issue, so that is nice to know.

by ultimabgd at November 22, 2019 01:35 AM

November 21, 2019


David Medeiros

Release 0.4 - More Github, More Problems

And so it continues....

After the spectacular disaster that was Release 0.3, I have returned this week for my fourth and final release. At this point, I am happy with everything I have done so far in this course, and these last few weeks will test everything I have learned so far. To start, I am going to go over some updates from last week's posts, and explain my plans for Release 0.4.

Release 0.3 Updates

I received some change requests from Flo as follows:
  • Make different grid sizes for different screen sizes
  • Try to use tailwind classes to attempt this
  • Change some syntax and naming
  • Adjust font size and justification
The conversation is listed here. So I successfully made these changes to the code aside from using tailwind. I couldn't get tailwind to work with the .module.css file type, as tailwind works better with regular css. I used some of the syntax to adjust font size and justification, but I couldn't the other way. I ended up using the @media method to detect screen size and adjust the grid to the required size. Some of the changes made also help with a compression issue, where some images were being squeezed to fit in the grid at certain screen sizes.

@media method

300px is for phone screens, 768px is for tablets, 992px is for smaller computer screens, and 1200px is for larger screens. Each of these has a different number of columns to fit the screen. 

Telescope - Release 0.4

I have picked an issue to work on for Release 0.4, to continue my work from 0.3 on documentation. My plan is to work on this issue to start, and then continue to monitor and update documentation as the Release continues. Part of my problem is that I have no way as far as I know to install docker in my current environment. I use WSL (Windows Subsystem for Linux), which cannot use docker, and I need Windows Professional to use docker desktop. I am stuck working on documentation and organization until I can get docker up and running. I am going to continue researching ways to get docker going over the weekend.

External PR - Release 0.4

I really enjoyed working on open climate fix this week, it made last week's madness feel a lot more legitimate. In terms of an external PR for Release 0.4, I really want to work on a page for a website. I would love to continue working on open climate fix, however there doesn't seem to be an escalated issue on that site at the moment, except for a contributing page which I am unsure of the complexity of. I am currently searching for an issue similar to what I have worked on these past 3 months. My criteria for a new issue are the following:
  • UI related (components, new pages, new features)
  • In an environment I am somewhat familiar with (Gatsby, React, Angular, Javascript, etc)
  • An escalation from my previous issue (a gatsby component with simple file reading)
Some issues that I have found that somewhat satisfy my criteria:

by drwm-base (noreply@blogger.com) at November 21, 2019 07:38 PM


Lexis Holman

Project Mile Marker

Hello again, I have another post about my attempt at optimizing clamav, which if you are new to my blog you can check out previous posts here:

https://penguins879376500.wordpress.com/category/school-work/project/

 

In my last post I was able to get into the source code for my project, and after what I feel was a considerable amount of analysis of a single function, I have started my optimization attempt. As of currently I am now getting the correct output on a single directory; add recursion into the mix and you get a segfault.

However, I have noticed a considerable speed up in the program, running against a single directory without recursion (because it’s broken) we have a run time of:

----------- SCAN SUMMARY -----------
Scanned directories: 1
Scanned files: 8
Infected files: 1
Data scanned: 8.00 MB
Data read: 2089.99 MB (ratio 0.00:1)
Time: 27.139 sec (0 m 27 s)
Start Date: 2019:11:20 16:25:02
End Date: 2019:11:20 16:25:30

In comparison to a previous run time of:

----------- SCAN SUMMARY -----------
Scanned directories: 1
Scanned files: 8
Infected files: 1
Data scanned: 8.00 MB
Data read: 2089.99 MB (ratio 0.00:1)
Time: 47.126 sec (0 m 47 s)
Start Date: 2019:11:20 16:23:36
End Date: 2019:11:20 16:24:23

Which is a significant improvement to the point that I am skeptical, especially with the fact that we cannot run this against a larger data set without a segfault. There are too many discrepancies to say for certain but the numbers are consistent and the output is correct; it at least gives me some hope for this project.

My next task of course is to solve that issue of the segfault which I believe relates to aligning the modified block to avoid loading values past the end of an array. But after fighting with the software to get the correct output when it comes to character strings makes me feel that making this post is required.

 

Modification overview:

The most difficult thing about this optimization was trying to get the character fields to load in at least 16 characters at a time without overrunning the buffer. The first run through our SIMD intrinsic operation went off without any issue and returned the index of the first not valid character, perfect. The second run, for any string that is longer than 16 characters (most of them) required that we “rewind” the current position pointer. The reason we need to do this is to make sure that we are getting 16 characters and not garbage values off the end of the buffer which could cause erroneous data.

That was completed with a ternery operator, to select the maximum number of characters and then subtracting the index by the number of characters to be loaded into the vector registers, it looked like this:

                L = p->prefix_length + p->length);

                J = (length > l) ? length -16 : l – 16;

This allows me to load the string from a safe offset; even though we are re checking some characters it is still much faster than checking our loop condition after every single character comparison.

From there we could load the values like:

                xbp = _mm_loadu_si128(__m128i*) &bp[j]);

                xpt = _mm_loadu_si128(__m128i*) &pt[j]);

Here is a graphic representation:

post_one

The green and yellow together are a complete string length, in this case 21 characters. The blue represents vector register lanes where each individual character can be loaded into and the yellow represents characters that are beyond the length of the registers or more then index 16 from the beginning of the array.

You can see that we still need to check the next 5 characters after the first call, however we cannot just load the vector register with then next set of 16 characters as we will load data off the end of the array as demonstrated in this image:

post_two

Now here the indigo color (that runs off the edge of the image) represents the characters in the string that have already been checked. The green represents the characters that are to be checked and then the red represents data that is past the end of our character array, essentially it is filled with arbitrary data. Attempting to access this data with either cause the program to segfault or worse, run checks against that data and corrupt our output. This is where the above solution comes into play with rewinding the pointer back to a point where we can check all the new characters, even if it means checking some of the previous ones; because we are doing this in one instruction it is much fasters then looping over every single character to determine a match. This alignment would look something like this:

post_three

Here we have indigo representing checked characters, yellow representing characters that are being rechecked and green being our next set of characters to check.

After we complete each check operation the intrinsic function that we are using returns an index that can be used to determine if a match on the entire string has been found. In either any indexes that rely on that information can then be set and seem to be running appropriately we are getting the proper output in terms of character string matches and now we can move on to the next part.

Determining a loop condition that can be used to allow this function to iterate through large blocks of non-text data which I believe is where we are causing our segfault, and the unexpected speed up.

 

Summery:

This post was only a quick update on my work so far with my chosen piece of software. Breaking past this milestone means that we (maybe) have a handle on how the offsetting is handled. Our next task it make sure that this function works with any byte array not only text. To do this we need to determine what data is causing it to crash and in what context, or function it is being called during the segfault. Once that is solved it is next a matter of cleaning up the code for efficiency and maintainability, but one thing at a time.

Hope to have more for you soon.

ElliePenguins.

by elliepenguins at November 21, 2019 02:01 PM


Weslie Chung

SPO 600 lab 3

What’s this?

For our third lab for SPO 600 (Software Portability and Optimization), I am to write code in assembly, the low level programming language mentioned in the last lab. There’s a lot to do so let’s get started.

Build in C

The following are 3 pieces of code that I will be building

hello.c

/* Hello World in traditional C using printf() */

#include

int main() {
printf(“Hello World!\n”);
}

hello2.c

/* Hello World with a direct write to stdout (file descriptor 1) */

#include

int main() {
write(1,”Hello World!\n”,13);
}

hello3.c

/* Hello World using a direct kernel system call to write to
file descriptor 1 (stdout) */

#include
#include

int main() {
syscall(__NR_write,1,”Hello World!\n”,13);
}

Hey, hello3.c has this thing call syscall. From what I can gather, it allows the application to directly do kernel services calls. Alright, let’s compile and see what’s underneath these different approaches with the help of objdump -d

hello

0000000000401126 :
401126: 55 push %rbp
401127: 48 89 e5 mov %rsp,%rbp
40112a: bf 10 20 40 00 mov $0x402010,%edi
40112f: b8 00 00 00 00 mov $0x0,%eax
401134: e8 f7 fe ff ff callq 401030
401139: b8 00 00 00 00 mov $0x0,%eax
40113e: 5d pop %rbp
40113f: c3 retq

hello2

0000000000401126 :
401126: 55 push %rbp
401127: 48 89 e5 mov %rsp,%rbp
40112a: ba 0d 00 00 00 mov $0xd,%edx
40112f: be 10 20 40 00 mov $0x402010,%esi
401134: bf 01 00 00 00 mov $0x1,%edi
401139: e8 f2 fe ff ff callq 401030
40113e: b8 00 00 00 00 mov $0x0,%eax
401143: 5d pop %rbp
401144: c3 retq
401145: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
40114c: 00 00 00
40114f: 90

hello3

0000000000401126 :
401126: 55 push %rbp
401127: 48 89 e5 mov %rsp,%rbp
40112a: b9 0d 00 00 00 mov $0xd,%ecx
40112f: ba 10 20 40 00 mov $0x402010,%edx
401134: be 01 00 00 00 mov $0x1,%esi
401139: bf 01 00 00 00 mov $0x1,%edi
40113e: b8 00 00 00 00 mov $0x0,%eax
401143: e8 e8 fe ff ff callq 401030
401148: b8 00 00 00 00 mov $0x0,%eax
40114d: 5d pop %rbp
40114e: c3 retq
40114f: 90 nop

Our hello programs get progressively more complex, hello2 has more lines of code than hello and hello3 has even more. Now we build in assembly for x86_64 and aarch64.

x86_64 and aarch64 assembly

The code:

hello-nasm.s (x86_64)

section .text
global _start

_start:
mov rdx,len ; message length
mov rcx,msg ; message location
mov rbx,1 ; file descriptor stdout
mov rax,4 ; syscall sys_write
int 0x80

mov rax,1 ; syscall sys_exit
int 0x80

section .rodata

msg db ‘Hello, world!’,0xa
len equ $ – msg

hello-gas.s (x86_64)

/*
This is a ‘hello world’ program in x86_64 assembler using the
GNU assembler (gas) syntax. Note that this program runs in 64-bit
mode.

CTyler, Seneca College, 2014-01-20
Licensed under GNU GPL v2+
*/

.text
.globl _start

_start:
movq $len,%rdx /* message length */
movq $msg,%rsi /* message location */
movq $1,%rdi /* file descriptor stdout */
movq $1,%rax /* syscall sys_write */
syscall

movq $0,%rdi /* exit status */
movq $60,%rax /* syscall sys_exit */
syscall

.section .rodata

msg: .ascii “Hello, world!\n”
len = . – msg

hello.s (aarch64)

.text
.globl _start
_start:

mov x0, 1 /* file descriptor: 1 is stdout */
adr x1, msg /* message location (memory address) */
mov x2, len /* message length (bytes) */

mov x8, 64 /* write is syscall #64 */
svc 0 /* invoke syscall */

mov x0, 0 /* status -> 0 */
mov x8, 93 /* exit is syscall #93 */
svc 0 /* invoke syscall */

.data
msg: .ascii “Hello, world!\n”
len= . – msg

This all reminds me of the <main> sections of that we’ve seen in my lab 2 post. There are also a lot more instructions compared to the pre-compiled C files. Let’s compile and objdump -d as well

hello-nasm.s (x86_64)

0000000000401000 :
401000: ba 0e 00 00 00 mov $0xe,%edx
401005: 48 b9 00 20 40 00 00 movabs $0x402000,%rcx
40100c: 00 00 00
40100f: bb 01 00 00 00 mov $0x1,%ebx
401014: b8 04 00 00 00 mov $0x4,%eax
401019: cd 80 int $0x80
40101b: b8 01 00 00 00 mov $0x1,%eax
401020: cd 80 int $0x80

hello-gas.s (x86_64)

0000000000401000 :
401000: 48 c7 c2 0e 00 00 00 mov $0xe,%rdx
401007: 48 c7 c6 00 20 40 00 mov $0x402000,%rsi
40100e: 48 c7 c7 01 00 00 00 mov $0x1,%rdi
401015: 48 c7 c0 01 00 00 00 mov $0x1,%rax
40101c: 0f 05 syscall
40101e: 48 c7 c7 00 00 00 00 mov $0x0,%rdi
401025: 48 c7 c0 3c 00 00 00 mov $0x3c,%rax
40102c: 0f 05 syscall

hello.s (aarch64)

00000000004000b0 :
4000b0: d2800020 mov x0, #0x1 // #1
4000b4: 100800e1 adr x1, 4100d0
4000b8: d28001c2 mov x2, #0xe // #14
4000bc: d2800808 mov x8, #0x40 // #64
4000c0: d4000001 svc #0x0
4000c4: d2800000 mov x0, #0x0 // #0
4000c8: d2800ba8 mov x8, #0x5d // #93
4000cc: d4000001 svc #0x0

Not only are all these fairly different from each other, they are also different from the C versions. However, these assembly codes all only have 8 lines, while two of the C versions have more lines, suggesting that assembly code is more efficient as less steps are needed to achieve the same result.

Writing loops in assembly

In this part of the lab, I’m to write a loop code in assembly to loop through the number from 0 to 9. While most of us are not taught how to code in assembly, we are given a piece of code that would loop 10 times, though nothing is in the body of the loop

.text
.globl _start

start = 0 /* starting value for the loop index; note that this is a symbol (constant), not a variable */
max = 10 /* loop exits when the index hits this number (loop condition is i<max) */

_start:
mov $start,%r15 /* loop index */

loop:
/* … body of the loop … do something useful here … */

inc %r15 /* increment index */
cmp $max,%r15 /* see if we're done */
jne loop /* loop if we're not */

mov $0,%rdi /* exit status */
mov $60,%rax /* syscall sys_exit */
syscall

So, after a lot of digging and looking through the commands/code used in assembly, I’ve come up with this:

.text
.globl _start

start = 0
max = 10
char = 48

_start:
mov $start,%r15
loop:
mov $len,%rdx
mov $msg,%rsi
mov $1,%rdi
mov $1,%rax
syscall

inc %r15
mov %r15,%r14
add $char,%r14
mov $ind,%r13
mov %r14b,(%r13)

cmp $max,%r15
jne loop

mov $0,%rdi
mov $60,%rax
syscall

.section .data

msg: .ascii “loop: 0\n”
.set len , . – msg
.set ind , msg + 6

The output from the code above:

So that was for x86_64, now let’s try to do that for aarch_64:

.text
.globl _start

start = 0
max = 10

_start:
mov x19,start
loop:
mov x2, len
adr x1, msg
mov x0, 1

add x24,x19,48
strb w24, [x1,len-3]
mov x8, 64
svc 0

add x19,x19,1
cmp x19, max
b.lt loop

mov x0, 0
mov x8, 93
svc 0

.section .data
msg: .ascii “Loop: \n”
len= . – msg

Remember how the ‘Hello world’ codes differ for both architectures? We had to account for those changes. The output are as follows:

There are 2 lines of difference in terms of line count between the two, with aarch64 2 fewer lines/operations.

Conclusion

Programming in assembly is tedious, one of my friends said that ever since he took an assembly course in high school, he swore to never program again, and I can see why. On top of having to understand something that’s close to machine code, having to keep track of registers to store and send data is quite a pain. A lot of high level programming languages deals with these issues for you and I will admit that spoiled by them, but I can see how low level languages could give you the edge you need when optimizing performance.

by Weslie Chung at November 21, 2019 05:55 AM

SPO 600 lab 2

What’s this?

For our second lab for SPO 600 (Software Portability and Optimization), I am to build a simple piece of code in a few different ways, and observe the changes in terms of optimization.

The code

#include

int main() {
printf(“Hello World!\n”);
}

This is perhaps one of the most famous piece of code, for those who are unfamiliar with programming, this would instruct the machine to output the message ‘Hello World!’ followed by inserting a new line (the equivalent to pressing return on your keyboard).

What am I doing?

The first thing I was instructed to do is to build the program with the options:

-g               # enable debugging information
-O0              # do not optimize (that's a capital letter and then the digit zero)
-fno-builtin     # do not use builtin function optimizations

The command I used to run to build was:

gcc lab1.c -g -O0 -fno-builtin -o lab1a

After building this, running

objdump -d lab1a

Would show us the assembly code (a low level language that works very close to the hardware, allows better customization to your code but makes it hard to be read by humans).

0000000000400594 :
400594: a9bf7bfd stp x29, x30, [sp, #-16]!
400598: 910003fd mov x29, sp
40059c: 90000000 adrp x0, 400000
4005a0: 9119c000 add x0, x0, #0x670
4005a4: 97ffffb7 bl 400480
4005a8: 52800000 mov w0, #0x0 // #0
4005ac: a8c17bfd ldp x29, x30, [sp], #16
4005b0: d65f03c0 ret
4005b4: 00000000 .inst 0x00000000 ; undefined

This is what the assembly code of what I wrote would look like.

More options!

The next part of this lab is to build the same code with a few different options, so let’s get started.

Add -static

gcc lab1.c -g -O0 -fno-builtin -static -o lab1b

One thing to note is that the result of this build option has a much high file size.

[wchung_rsa@aarchie lab1]$ gcc lab1.c -g -O0 -fno-builtin -static -o lab1b
[wchung_rsa@aarchie lab1]$ ll
total 596
-rwxrwxr-x. 1 wchung_rsa wchung_rsa 73144 Nov 20 20:35 lab1a
-rwxrwxr-x. 1 wchung_rsa wchung_rsa 631712 Nov 20 20:51 lab1b
-rw-rw-r–. 1 wchung_rsa wchung_rsa 65 Nov 20 20:27 lab1.c

More than 10 times the size, so what does the -static option do? According to the documentation it says that ‘On systems that support dynamic linking, this overrides -pie and prevents linking with the shared libraries. On other systems, this option has no effect.’

Modern programs uses shared libraries when they run, having the -static option would create a build where the program wouldn’t have to rely on any libraries when running by inserting it to the file, the linking with the libraries needed for the program would be static, hence the name.

Remove -fno-builtin

gcc lab1.c -g -O0 -o lab1c

Size wise, this change doesn’t seem to have much of an effect in this respect

[wchung_rsa@aarchie lab1]$ gcc lab1.c -g -O0 -o lab1c
[wchung_rsa@aarchie lab1]$ ll
total 628
-rwxrwxr-x. 1 wchung_rsa wchung_rsa 73144 Nov 20 20:35 lab1a
-rwxrwxr-x. 1 wchung_rsa wchung_rsa 631712 Nov 20 20:51 lab1b
-rw-rw-r–. 1 wchung_rsa wchung_rsa 65 Nov 20 20:27 lab1.c
-rwxrwxr-x. 1 wchung_rsa wchung_rsa 73128 Nov 20 21:15 lab1c
-rw-r–r–. 1 wchung_rsa wchung_rsa 16384 Nov 20 21:10 ”

So how does it differ from not removing that option? The -fno-builtin option will have the build not use optimizations for built in functions. The un-optimized version of the code uses the printf function, while the optimized build uses the puts function. The difference between these two functions is that puts only displays strings, while printf can display many different types of variable values. This is also why the un-optimized version has more commands, as it needs to set the output type for the printf command.

Remove -g

gcc lab1.c -O0 -fno-builtin -o lab1d

This build is a bit smaller than other similar builds

[wchung_rsa@aarchie lab1]$ ll
total 644
-rwxrwxr-x. 1 wchung_rsa wchung_rsa 73144 Nov 20 21:27 lab1a
-rwxrwxr-x. 1 wchung_rsa wchung_rsa 631712 Nov 20 20:51 lab1b
-rw-rw-r–. 1 wchung_rsa wchung_rsa 65 Nov 20 20:27 lab1.c
-rwxrwxr-x. 1 wchung_rsa wchung_rsa 73128 Nov 20 21:25 lab1c
-rwxrwxr-x. 1 wchung_rsa wchung_rsa 70640 Nov 20 21:40 lab1d
-rw-r–r–. 1 wchung_rsa wchung_rsa 16384 Nov 20 21:10 ”

The -g option generates debug info for the build, hence the increased size.

Add additional arguments to printf()

The is how I changed my primitive code

#include

int main() {
printf(“Hello World! test test test\n %d”, 1);
}

gcc lab1more.c -g -O0 -fno-builtin -o lab1e

Size wise, there isn’t much of a difference.

But there are a good few differences in their assembly code:

More:
0000000000400594 :
400594: a9bf7bfd stp x29, x30, [sp, #-16]!
400598: 910003fd mov x29, sp
40059c: 52800021 mov w1, #0x1 // #1
4005a0: 90000000 adrp x0, 400000
4005a4: 9119c000 add x0, x0, #0x670
4005a8: 97ffffb6 bl 400480
4005ac: 52800000 mov w0, #0x0 // #0
4005b0: a8c17bfd ldp x29, x30, [sp], #16
4005b4: d65f03c0 ret

Original
0000000000400594 :
400594: a9bf7bfd stp x29, x30, [sp, #-16]!
400598: 910003fd mov x29, sp
40059c: 90000000 adrp x0, 400000
4005a0: 9119c000 add x0, x0, #0x670
4005a4: 97ffffb7 bl 400480
4005a8: 52800000 mov w0, #0x0 // #0
4005ac: a8c17bfd ldp x29, x30, [sp], #16
4005b0: d65f03c0 ret
4005b4: 00000000 .inst 0x00000000 ; undefined

The line: 40059c: 52800021 mov w1, #0x1 // #1 was where differ greatly.

Move the printf() call to a separate function named output()

#include

void output(){
printf(“Hello World!\n”);
}

int main(){
output();
}

gcc lab1diff.c -g -O0 -fno-builtin -o lab1f

The differences in their assembly code:

With output()

0000000000400594 :
400594: a9bf7bfd stp x29, x30, [sp, #-16]!
400598: 910003fd mov x29, sp
40059c: 90000000 adrp x0, 400000
4005a0: 911a2000 add x0, x0, #0x688
4005a4: 97ffffb7 bl 400480
4005a8: d503201f nop
4005ac: a8c17bfd ldp x29, x30, [sp], #16
4005b0: d65f03c0 ret

00000000004005b4 :
4005b4: a9bf7bfd stp x29, x30, [sp, #-16]!
4005b8: 910003fd mov x29, sp
4005bc: 97fffff6 bl 400594
4005c0: 52800000 mov w0, #0x0 // #0
4005c4: a8c17bfd ldp x29, x30, [sp], #16
4005c8: d65f03c0 ret
4005cc: 00000000 .inst 0x00000000 ; undefined

Original

0000000000400594 :
400594: a9bf7bfd stp x29, x30, [sp, #-16]!
400598: 910003fd mov x29, sp
40059c: 90000000 adrp x0, 400000
4005a0: 9119c000 add x0, x0, #0x670
4005a4: 97ffffb7 bl 400480
4005a8: 52800000 mov w0, #0x0 // #0
4005ac: a8c17bfd ldp x29, x30, [sp], #16
4005b0: d65f03c0 ret
4005b4: 00000000 .inst 0x00000000 ; undefined

The output() version of <main> had less lines, but they have many similar lines. The original code had two more lines than the output() version’s <main>. But, those missing lines are found in the output() versions’s <output> section, which makes sense as the output function calls for the same operation executed in the original version.

Remove -O0 and add -O3

gcc lab1.c -g -O3 -fno-builtin -o lab1g

Let’s look at their assembly code:

O3

0000000000400490 :
400490: a9bf7bfd stp x29, x30, [sp, #-16]!
400494: 90000000 adrp x0, 400000
400498: 9119c000 add x0, x0, #0x670
40049c: 910003fd mov x29, sp
4004a0: 97fffff8 bl 400480
4004a4: 52800000 mov w0, #0x0 // #0
4004a8: a8c17bfd ldp x29, x30, [sp], #16
4004ac: d65f03c0 ret

Original

0000000000400594 :
400594: a9bf7bfd stp x29, x30, [sp, #-16]!
400598: 910003fd mov x29, sp
40059c: 90000000 adrp x0, 400000
4005a0: 9119c000 add x0, x0, #0x670
4005a4: 97ffffb7 bl 400480
4005a8: 52800000 mov w0, #0x0 // #0
4005ac: a8c17bfd ldp x29, x30, [sp], #16
4005b0: d65f03c0 ret
4005b4: 00000000 .inst 0x00000000 ; undefined

the -O(s) options are for compiler optimization, and for our humble code, the -O3 option took out the

400598: 910003fd mov x29, sp

line, my assumption is that the removed line unnecessary, and thus the compiler decided to remove it.

Conclusion

Modern compilers now include a lot of options to potentially improve a build’s performance, some of these improvements are at the machine level (assembly) and at times, it is best to leave these minor optimizations for the compiler to make.

by Weslie Chung at November 21, 2019 03:21 AM

SPO 600 lab 1

What’s this?

For our first lab for SPO 600 (Software Portability and Optimization), I am to look for two open source software packages with different licenses, and talk about how contributors submit their changes, as well as an example from beginning to finish of a successful software change.

Open source what?

If you are not familiar with open source software development, the layman explanation of how they work is that the developer(s) of a piece of open source software would usually upload their source code (blue prints of a software) onto an online software repository site, such as Github to allow other developers to view and edit their code, any changes are however verified by members part of the community of the developed software. It’s a great way to get help with your project, and you’d be helping the software development scene in general whether you’ve got your own project, or you’re contributing to one.

What did I pick?

For this lab, I have to pick two projects with different open source licenses. These licenses are what ‘protects’ these projects, they all more or less states that anyone may edit and use whatever code they find in this project, provided that they’d have to also attach this license to their project as well, and that any projects that uses any part of an open source project may not be used commercially.

With that our of the way, I’ve picked Sourcetrail and Flutter. They use the GNU and chromium license respectively. Both of these projects are hosted on Github.

Sourcetrail

Based on this pull request, this project seems to have a somewhat short response rate of about 2 hours, it however lacks any reviewers.

Flutter

Based on this pull request, this project has an even shorter response rate, a reviewer was requested and approved the changes. Also, a few labels were added by bots, perhaps to better categorize what sort of changes this request involved.

Conclusions

Flutter seemed like it had a more developed community, and indeed, flutter’s contribution guide contains a more fleshed-out guideline than sourcetrail’s guide. Both projects openly states that they welcome contributors, and with how it looks to me flutter’s strict and precise rules of engagement would benefit a veteran, and sourcetrail would make for a nice ‘beginner’ open source project for those who are new to the scene to contribute to.

by Weslie Chung at November 21, 2019 01:17 AM

November 20, 2019


Bowei Yao

D-link DWA-182 driver for Ubuntu 18.04

My wifi mysteriously died in Ubuntu 18.04 for my dual-boot desktop, again. Just a quick heads-up, my desktop does not have a built-in wifi adapter. I purchased a USB dongle a couple years ago, namely the D-link DWA-182 version C to enable it to connect to wifi. It died because linux kernel has been updated from Ubuntu 16.04 to 18.04, and my old driver no longer builds properly.

Here’s what I did to fix it:

git clone https://github.com/gnab/rtl8812au

This particular version: https://github.com/gnab/rtl8812au is updated to compile against kernel 5.2.

After getting the code, navigate to the repository, then run:

sudo make
sudo cp 8812au.ko /lib/modules/$(uname -r)/kernel/drivers/net/wireless
sudo depmod -a

by Schrodinger's Cat at November 20, 2019 09:54 PM


Hayley McIldoon

Better Late Than Never

I am back! And it’s time to discuss release 0.3.

I am working on Telescope, a tool for tracking blogs about open source at Seneca. I came to the game a bite late, so finding an issue to file or work on was a bit daunting. Thankfully my open source teacher was able to help me select an issue to work on.

The issue I selected was to sanitize HTML from feed posts. What does “sanitize HTML” even mean you ask? It takes code that looks like this: <img src=x onerror=alert(1)//> and turns it into code that looks like this: <img src=”x”> preventing XSS attacks (malicious scripts injected into a website) from happening.

My teacher suggested I look into DOMPurify, a JavaScript library that sanitizes HTML. Fortunately for me, the DOMPurify documentation is easy to understand and I had another students work (jatinAroraGit) to reference to help me get started.

The code itself isn’t too exciting, but if you are curious here it is:

const createDOMPurify = require(‘dompurify’);
const { JSDOM } = require(‘jsdom’);

module.exports.run = async function(text) {
const { window } = new JSDOM();
const DOMPurify = createDOMPurify(window);
return Promise.resolve(DOMPurify.sanitize(text));
};

The biggest issues I had were remembering git commands (very easy to forget if you aren’t doing them daily!) and reminding myself how tests work again. In all of my classes at Seneca I have rarely, if at all, worried about testing. The further along in the CPA program I get and the more interviews I have, I can see testing is very important and is often overlooked. I’m happy for this release we are being asked to create tests to make us accountable for our code.

For my external project I am back to working on Home Assistant’s back-end in Python. I had started working on device conditions last release. The Home Assistant team is waiting to release the new version of Home Assistant but they are waiting for device conditions to be completed. It looks like most of the contributors are too busy at the moment, so I have stepped in and have started to complete them so I can help the team release the new version of Home Assistant sooner.


by hmcildoon at November 20, 2019 09:02 PM


Bowei Yao

Some algorithms

I applied for a job at Amazon a while ago, and failed the online AMCAT test. I can’t remember the exacts of this one given problem, but more or less it’s something similar to this one:

Given 2 lists of numbers and a maximum, find the maximum of sum of a number from list1 + a number from list2 that is smaller or equal to the given maximum, as well as the number of occurrences.

Since I failed and didn’t pass the test suite given by Amazon, I decided to spend some time hacking away and tinkering with algorithms here: https://github.com/ragnarokatz/algorithm. My goal here is to create a more efficient algorithm to solve this problem, because quite possibly one of the reasons I didn’t pass the test suite is that the amount of time required to compute exceeded the amount of time allowed by AMCAT.

Here’s the pure bash version:

import time
import random


start_time = time.time()
random.seed(0)

ROUTES_TO = [random.randint(0, 10000) for n in range(10000)]
ROUTES_BACK = [random.randint(0, 10000) for n in range(10000)]
MAX = 10000

count = 0
current_max = 0

for i in range(len(ROUTES_TO)):
    for j in range(len(ROUTES_BACK)):
        total = ROUTES_TO[i] + ROUTES_BACK[j]
        if total > MAX:
            continue
        if total > current_max:
            current_max = total
            count = 1
        elif total == current_max:
            count += 1

end_time = time.time()

print (f'upper limit set to = {MAX}')
print (f'current maximum possible = {current_max}, number of combinations = {count}')
print( f'time elapsed = {round(end_time - start_time, 2)}')

I’ve set the random seed to be the same in every single case in order to come up with the exact same answers in all the algorithms. The max is set to 10000. The first version is just a double loop through the lists, recording a new maximum whenever encountered. It takes about 25.28s to finish on my computer.

Improved version, sorted from small to large:

import time
import random


start_time = time.time()
random.seed(0)

MAX = 10000
ROUTES_TO = [random.randint(0, 10000) for n in range(10000)]
ROUTES_BACK = [random.randint(0, 10000) for n in range(10000)]

ROUTES_TO.sort()
ROUTES_BACK.sort()

count = 0
current_max = 0


for i in range(len(ROUTES_TO)):
    for j in range(len(ROUTES_BACK)):
        total = ROUTES_TO[i] + ROUTES_BACK[j]
        if total > MAX:
            break
        if total > current_max:
            current_max = total
            count = 1
        elif total == current_max:
            count += 1

end_time = time.time()

print (f'upper limit set to = {MAX}')
print (f'current maximum possible = {current_max}, number of combinations = {count}')
print( f'time elapsed = {round(end_time - start_time, 2)}')

The second version sorts the lists in an ascending fashion, then starts iterating through with the smallest. Whenever a sum is detected to exceed the maximum, it breaks the current iteration. Since if a number that is smaller than the next one already exceeds the maximum, then surely the next number will exceed the maximum as well. This one took 14.56s.

Even better version: sorted from large to small:

import time
import random


start_time = time.time()
random.seed(0)

MAX = 10000
ROUTES_TO = [random.randint(0, 10000) for n in range(10000)]
ROUTES_BACK = [random.randint(0, 10000) for n in range(10000)]

ROUTES_TO.sort(reverse=True)
ROUTES_BACK.sort(reverse=True)

count = 0
current_max = 0


for i in range(len(ROUTES_TO)):
    for j in range(len(ROUTES_BACK)):
        total = ROUTES_TO[i] + ROUTES_BACK[j]
        if total > MAX:
            continue
        if total < current_max:
            break
        elif total == current_max:
            count += 1
        else:
            current_max = total
            count = 1

end_time = time.time()

print (f'upper limit set to = {MAX}')
print (f'current maximum possible = {current_max}, number of combinations = {count}')
print( f'time elapsed = {round(end_time - start_time, 2)}')

This version sorts the lists in descending fashion first. Starting with the biggest sum that is smaller than the given maximum, it will break the current iteration if a smaller sum is detected. If a maximum has already been produced and the next number is smaller, then surely the sum will not exceed the current maximum. It took 8.89s to finish.

This is all I can come up with at the moment off the top of my head. I’m planning on looking into some dynamic programming ideas and see if I can incorporate any of those into the current solution. Here’s a useful link if you are interested: https://www.youtube.com/watch?v=vYquumk4nWw

by Schrodinger's Cat at November 20, 2019 04:49 AM

November 19, 2019


Yiran Zhu

SPO600 SIMD Lab 5 (Part 1)

Purpose of this Lab
In this lab, you will investigate the use of SIMD instructions in software, using auto-vectorization, inline assembler, and C intrinsics.

https://wiki.cdot.senecacollege.ca/wiki/SPO600_SIMD_Lab

What is SIMD?

SIMD, acronym for Single Instruction, Multiple Data; “refers to a class of instructions which perform the same operation on several separate pieces of data in parallel” (Tyler).

The purpose of learning SIMD capabilities in this lab is to utilize it in three different ways:

  • Auto-Vectorization: adding compiler options to vectorize loops, automatically generating SIMD code
  • Inline Assembler: adding architecture-specific assembly language embedded in C programs, explicitly including SIMD instructions
  • C Intrinsics: adding function-like extensions, groups of intrincics which provide access to SIMD instructions

Instructions

Part 0: Setup

Unpack the archive to home directory:

[yzhu132@aarchie ~]$ mkdir lab5
[yzhu132@aarchie ~]$ tar -zxvf /public/spo600-simd-lab.tgz -C ~/lab5/
spo600/
spo600/simd_lab/
spo600/simd_lab/vol1.c
spo600/simd_lab/vol_intrinsics.c
spo600/simd_lab/vol_inline.c
spo600/simd_lab/vol.h
spo600/simd_lab/Makefile
spo600/simd_lab/add.c

Part 1: Auto-Vectorization

Modify the Makefile so that this file is compiled with the option -fopt-info-vec-all:

[yzhu132@aarchie ~]$ cd lab5/spo600/simd_lab/
[yzhu132@aarchie simd_lab]$ ls -l
total 24
-rw-r--r--. 1 yzhu132 yzhu132  351 Oct 11 13:09 add.c
-rw-r--r--. 1 yzhu132 yzhu132  393 Oct  3 13:19 Makefile
-rw-------. 1 yzhu132 yzhu132 1007 Oct  2 12:57 vol1.c
-rw-r--r--. 1 yzhu132 yzhu132   24 Oct  2 09:33 vol.h
-rw-r--r--. 1 yzhu132 yzhu132 2225 Oct  2 09:30 vol_inline.c
-rw-r--r--. 1 yzhu132 yzhu132 1577 Oct  2 09:20 vol_intrinsics.c
[yzhu132@aarchie simd_lab]$ nano Makefile 
BINARIES = vol_inline vol_intrinsics add vol1
CCOPTS = -g -O3 
AUTOVECTOROPTS = -fopt-info-vec-all
CC=gcc

all:            ${BINARIES}

vol_inline:     vol_inline.c vol.h
                ${CC} ${CCOPTS} vol_inline.c -o vol_inline

vol_intrinsics: vol_intrinsics.c vol.h
                ${CC} ${CCOPTS} vol_intrinsics.c -o vol_intrinsics

vol1:           vol1.c vol.h
                ${CC} ${CCOPTS} vol1.c -o vol1

add:            add.c
                ${CC} ${CCOPTS} add.c -o add

clean:  
                rm ${BINARIES} || true

auto_vector:    vol1.c vol.h
                ${CC} ${CCOPTS} ${AUTOVECTOROPTS} vol1.c -o vol1

Now compile vol1.c and review the compiler output. By running the following command, it saves the output into a text file:

[yzhu132@aarchie simd_lab]$ make auto_vector &> auto_out.txt
[yzhu132@aarchie simd_lab]$ less auto_out.txt 

Search for lines which has “vectorized” by running this in less:

/vectorized

Found the following blocks of lines:

...
vol1.c:32:2: note: loop vectorized
vol1.c:32:2: note: === vec_transform_loop ===
vol1.c:32:2: note: ------>vectorizing phi: x_52 = PHI <x_35(10), 0(12)>
vol1.c:32:2: note: ------>vectorizing phi: .MEM_56 = PHI <.MEM_34(10), .MEM_31(12)>
vol1.c:32:2: note: ------>vectorizing phi: ivtmp_75 = PHI <ivtmp_74(10), 5000000(12)>
vol1.c:32:2: note: ------>vectorizing statement: # DEBUG x => x_52
vol1.c:32:2: note: ------>vectorizing statement: # DEBUG BEGIN_STMT
...
...
vol1.c:38:2: note: not vectorized: not enough data-refs in basic block.
vol1.c:38:2: note: ===vect_slp_analyze_bb===
vol1.c:38:2: note: ===vect_slp_analyze_bb===
vol1.c:43:2: note: === vect_analyze_data_refs ===
vol1.c:43:2: note: not vectorized: not enough data-refs in basic block.
vol1.c:43:2: note: === vect_analyze_data_refs ===
vol1.c:43:2: note: not vectorized: not enough data-refs in basic block.
...

Looks like only one of the two loops was vectorized, and it’s the loop at line 32.

To vectorize the other loop, I’ll have to remove the modulus operation in line 39 from vol1.c.

[yzhu132@aarchie simd_lab]$ vi vol1.c 

Before:

...
        // Sum up the data
        for (x = 0; x < SAMPLES; x++) {
                ttl = (ttl+data[x])%1000;
        }
...

After:

...
        // Sum up the data
        for (x = 0; x < SAMPLES; x++) {
                ttl = (ttl+data[x]);
        }
...

Time to remove the previous vol1 out file and remake vol1.

[yzhu132@aarchie simd_lab]$ rm vol1  
[yzhu132@aarchie simd_lab]$ make auto_vector &> auto_out_vectorized.txt
[yzhu132@aarchie simd_lab]$ less auto_out_vectorized.txt

And now when searching for “vectorized”, we see that the loop in line 38 is now vectorized:

...
vol1.c:38:2: note: loop vectorized
vol1.c:38:2: note: === vec_transform_loop ===
vol1.c:38:2: note: ------>vectorizing phi: x_52 = PHI <x_36(9), 0(12)>
vol1.c:38:2: note: ------>vectorizing phi: ttl_53 = PHI <ttl_35(9), 0(12)>
vol1.c:38:2: note: multiple-types.
vol1.c:38:2: note: transform phi.
...

by naiuhz at November 19, 2019 08:14 PM


Daniel Ameresekere

SPO600 Project (Part 3) Benchmarking and Profiling

Now that I have selected the Barcode Project, I have started to benchmark it. Using both Aarch64 and x86_64, I have gone through some files to try to understand the code of this project. I built and tested the project and the outcome was:

The above is the output of the barcodes being printed. For example this shows a barcode called “silly code” being printed. It gives the scale number and is encoded as Code 39. Every barcode has a different set of lines and spaces with various widths that creates specific patterns such as Code 39 as mentioned above, Code 128, Codabar, Plessey, etc. After that print message, you’ll see a series of number that represents height, position(X-axis), position(Y-axis) and width. These four creates the shape and position of the barcode.

After getting the shape and position of the barcode, the characters of the barcode (in this example, “silly code”) are assigned positions and font size so they can printed below (or sometimes above depending on the type) the barcode. The image below shows the visual representations of the barcodes that were created:

BARCODE RUNTIME

After testing the project, I wanted to check on the project’s runtime. I displayed the runtime on both X86_64 and Aarch64 and it turned out to be a problem:

Runtime on X86_64
Runtime on Aarch64

The images above show the runtime of the project running on X86_64 and Aarch64. The real time to run the project on X86_64 was 0.010s and on Aarch64 was 0.028s, which were both too fast to analyze and the Professor suggested to increase the number of pages so that the runtime is increased. I have looked into this and have discussed it with some of my fellow classmates, and I will post my findings in the next blog post. Thanks for reading.

by danielameresekere at November 19, 2019 12:18 AM

November 18, 2019


Yiran Zhu

SPO600 Stage 1

Choosing a Project Again

Continuing from my last blog post, I’ve switch to FFmpeg due to problems with libav’s inactive community.

For this project, my goal is to optimize certain libraries and/or functions utilized by FFmpeg when it is used to burn SubRip Text (.srt) subtitles onto MPEG-4 Part 14 (.mp4) videos.

The following steps were done on both aarchie and bbetty, but most of the steps shown will be on aarchie.

First step, fork and clone the FFmpeg Github repo:

[yzhu132@aarchie ~]$ git clone git@github.com:naiuhz/FFmpeg.git
[yzhu132@aarchie ~]$ cd FFmpeg/
[yzhu132@aarchie FFmpeg]$

Building & Installation

Next step is to check the instructions in the INSTALL.md:

[yzhu132@aarchie FFmpeg]$ cat INSTALL.md
## Installing FFmpeg

1. Type `./configure` to create the configuration. A list of configure
options is printed by running `configure --help`.

    `configure` can be launched from a directory different from the FFmpeg
sources to build the objects out of tree. To do this, use an absolute
path when launching `configure`, e.g. `/ffmpegdir/ffmpeg/configure`.

2. Then type `make` to build FFmpeg. GNU Make 3.81 or later is required.

3. Type `make install` to install all binaries and libraries you built.

NOTICE
------

 - Non system dependencies (e.g. libx264, libvpx) are disabled by default.

Looks like I can run the configure in another directory so let’s try that:

[yzhu132@aarchie FFmpeg]$ mkdir build
[yzhu132@aarchie FFmpeg]$ cd build/
[yzhu132@aarchie build]$ /home/yzhu132/FFmpeg/configure CFLAGS="-g -pg"
Unknown option "CFLAGS=-g".
See /home/yzhu132/FFmpeg/configure --help for available options.

Huh, turns out this configure script doesn’t use the default variable syntax.

After calling “/home/yzhu132/FFmpeg/configure –help”, I read through the available help options and figured out the equivalent CFLAG variables:

[yzhu132@aarchie build]$ /home/yzhu132/FFmpeg/configure --enable-libass --extra-cflags="-g -pg" --extra-cxxflags="-g -pg"
ERROR: libass not found using pkg-config

If you think configure made a mistake, make sure you are using the latest
version from Git.  If the latest version fails, report the problem to the
ffmpeg-user@ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "ffbuild/config.log" produced by configure as this will help
solve the problem.

I included the option “–enable-libass” because FFmpeg requires the libass library to use the subtitles filter.

Draw subtitles on top of input video using the libass library. This filter requires ffmpeg to be compiled with –enable-libass.

https://trac.ffmpeg.org/wiki/HowToBurnSubtitlesIntoVideo

So I went ahead and installed libass’ dev libraries:

$ sudo dnf install libass-devel

And I reran the configure with the options in build.

[yzhu132@aarchie build]$ /home/yzhu132/FFmpeg/configure --enable-libass --extra-cflags="-g -pg" --extra-cxxflags="-g -pg"

Then, I ran make with the -j option on aarchie and bbetty with appropriate -j values.

[yzhu132@aarchie build]$ make -j 25
[yzhu132@bbetty build]$ make -j 9

Benchmarking

Now it’s time to burn subtitles onto a video. The video I chose to hardsub was CGP Grey’s Copyright: Forever Less One Day.

So I downloaded the video as a .mp4 file and the subtitles as .srt, and stored them in my Films directory on aarchie and bbetty.

[yzhu132@aarchie ~]$ tree Films/
 Films/
 ├── Copyright_Forever.mp4
 ├── Copyright_Forever.srt

0 directories, 2 files

In order to burn the subtitles into the videos, I ran this command:

[yzhu132@aarchie Films]$ /home/yzhu132/FFmpeg/build/ffmpeg -i Copyright_Forever.mp4 -filter:v subtitles=Copyright_Forever.srt Copyright_Forever_HC.mp4

And after a period of time, it produced the hardsub “Copyright_Forever_HC.mp4”.

Profiling

Finally, it’s profiling time. Unfortunately, I encountered difficulty using gprof to profile FFmpeg so I resorted to using perf instead. To profile with perf, I ran perf with the record argument along with the path of FFmpeg. I also added “–freq=max” “to interrupt the program to record analysis as many times as possible”. I got this idea from reading Lexis’ blog post which can be found here: https://penguins879376500.wordpress.com/2019/11/10/main-project-benchmark/.

[yzhu132@aarchie Films]$ perf record --freq=max -g /home/yzhu132/FFmpeg/build/ffmpeg 
info: Using a maximum frequency rate of 16,200 Hz
ffmpeg version N-95714-g20c5f4d835 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8 (GCC)
  configuration: --enable-libass --extra-cflags='-g -pg' --extra-cxxflags='-g -pg'
  libavutil      56. 35.101 / 56. 35.101
  libavcodec     58. 61.100 / 58. 61.100
  libavformat    58. 34.101 / 58. 34.101
  libavdevice    58.  9.100 / 58.  9.100
  libavfilter     7. 66.100 /  7. 66.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.030 MB perf.data (231 samples) ]

To view the performance date, run perf with the report argument:

$ perf report

The screenshot above is the result of running “perf report” on both aarchie and bbetty after burning the subtitles into a video.

Analysis

Looking at the right most column “Symbol”, the routines with the highest children overhead were:

  • _start
  • _dl_start
  • _dl_start_final
  • _dl_sysdep_start
  • dl_main
  • _dl_relocate_object
  • _dl_lookup_symbol_x
  • do_lookup_x
  • strcmp/check_match

The reason why the sum of the overhead values exceed 100% is because “each of them is already an accumulation of ‘self’ overhead of its child functions”. Further reading about overhead calculation can be found here: https://perf.wiki.kernel.org/index.php/Tutorial#Overhead_calculation

Stage 1 Conclusion

From analyzing the top five names of the routines over 90% overhead, 4/5 of them have the word “start” in them. This gives me a rough idea on what probably uses the most overhead in FFmpeg when burning subtitles into videos.

My optimization strategy is to first analyze the code in the routines with overhead over 10%. Methods I’ll be using are likely in-line assembler, compiler code optimization and/or algorithm improvements.

Stay tuned for my next blogs where I’ll attempt to use OProfile to profile from a different perspective and investigate the mysterious five routines with over 90% overhead.

by naiuhz at November 18, 2019 03:45 PM


Thomas Luu

Note to Self: AngularJS Isn’t Angular

So I continued working in the open source community and I decided to make further contributions to a previous repo I worked on, pagermon. Last time, I added some additional settings to the webpage that allowed for the toggling of each column in the pager table. My PR also added the ability to change the column names. The owners liked my previous PR but will not be merging the feature until they figure out how to fix the stylesheet so that it is consistent with my changes. This time, I took on an issue that required me to make a pop up modal for the messages table so that phones may have an easier time reading messages.

The feature is just the implementation of a modal in AngularJS. Note the keywords in the previous sentence: modal and AngularJS. It’s been a while since I have touched Angular and I completely forgot that AngularJS and Angular are two different things. So for the first iteration of the feature I wrote the code in angular and was honestly stumped when I couldn’t compile the code properly. Took me like an hour before I realized the code was written in AngularJS. Honestly, I could have saved a lot more time if I just read the documentation for the project, as it states it uses AngularJS. The sources code was also a big red flag as the syntax for Angular and AngularJS are different. But in my defence, I was lacking sleep and food as I am just a poor student.

When I figured out I was using the wrong framework, implementing the feature became a lot easier. Adding new components in AngularJS is quite simple, especially since the pagermon utilitzes Angular directives for Boostrap which provides base code that I can easily copy and modify in the documentation. The feature turned out nicely and the owners liked it, which made me quite satisfied.

by trusolz at November 18, 2019 03:25 PM


Ngoc Nam Khang Nguyen

Release 0.3b: #translation #maintainer - An amazing story with Gatsby.js

I have been imagining that I am going to have a good blog post for the following satisfactory and unforeseen story.

Initial Failure

Since last week, I have been very tempted to level-up my game with the contributions I have been making to open source. I planned to work on a project of Microsoft which turned out to be a spectacular failure for me mainly because a) I cannot grasp enough the complexity of the codebase to accomplish a good-enough contribution for my goal and b) many smaller issues were CSS modification when I was more interested in making contributions to functionality. However, that fact changed just the next day...

How Gatsby and I crossed each other's path

The following day, as my friends and I are staying after class, as usual, by chance, we had a conversation about how we are catching up with the heavy load of the semester and our open source contribution goal. A good friend of mine suddenly reminded me of a conversation we had before (when my interest was "functionality contribution").

Turned out, my friend had already signed up for a Spanish translation of Gatsby's documentation. That sparked me an idea! If she can do that, why can't I contribute to translating Gatsby's documentation into my language, too? After all, helping people in my country have a higher chance of approaching new technologies is kinda my goal, too... as well as getting free swag from Gatsby, of course (if you followed me on the Hacktoberfest series, you know how much I am obsessed with free swag).

However, my experience proved that the beginning is always the hardest part

Phase #1: Gatsby Vietnamese Translation is not a thing yet

I soon realize that there has been no ongoing effort to translate Gatsby.js to Vietnamese. I was looking around, even into other projects such as React translation progress where they achieved 55% of Vietnamese translation already.

I was really tempted to switch to React and was discouraged (since I won't get my free swag with React). That was until my friend, once again, told me: "Hey! You can be the first one doing it. Just request a new language translation!".

I was so happy and surprised that it was even a thing that I could be the initiator for Vietnamese translation. It also shocked me that "Hey, I might have to be a maintainer for the new repository.".

Phase #2: Looking for help

I rush myself to file an issue for the new translation request just to find out that I need help from others.

I admit that I cannot do this alone because...
  1. The number of documents in need of translation is enormous!
  2. Gatsby maintainer team requires at least 2 maintainers per translation repository so we can review each other's works.
  3. To be honest, I feel that I really need another set of eyes since I have not spent a day using Vietnamese technology language, just reading materials here and there. And I have not used Vietnamese as my primary language for 2 years so... yea, I have a problem with my linguistic sense of confidence.
So, I looked around for help, reached out to some of my Vietnamese friends. Some of them are back home, one or two are in the US and many are here, in Canada.

I talked to my professor for help reaching out to Vietnamese students in the class (and surprisingly got the attention of a Gatsby's maintainer through one of his tweets).

 
I jumped on the Gatsby Discord Server (the platform they use for community interaction) to spread the words about the new translation.


 Finally, 2 days of looking around and messaging back and forth paid off! I got 2 friends agreed to help me. One is a 3rd-semester student in my same program (Software Development, BSD) with whom I was fortunate to get acquainted through a mentoring program at Seneca College. The other one is my friend from middle school who is now going to college, majored in Computer Science, in the US.

I went ahead and filed my translation request. I finally got my translation maintainer team!

Phase #3: Waiting, waiting and more waiting for approval

After filing the issue is another story. The waiting feels endless to me since, well, I have a deadline to meet.

I waited for 48 hours (the average amount of time for previous translation requests to be accepted, as I noticed) before I ran out of patience and started contacting maintainers on the Discord Server.

Turned out that the person who was in charge of localization of Gatsby has been taking a break and they have been kinda shorted on personnel. However, I also found out who to contact to get my request approved.

Phase #4: Let's get things moving

After a productive conversation, I was so delighted to see my request approved, the repository for Vietnamese translation created and I can get started with my contribution.

 
I have already made my first PR to add a Vietnamese version of README.md file to support easy navigation for potential contributors. Since the maintainers of Gatsby do not allow changes to be made directly to the master branch, I talked to my fellow localization maintainer to review my PR and got it merged.

Hey! And just like that, I got my first Gatsby swag!

Looking forward

Something tells me that I will stick with this project for a long time (since, of course, I want more swag). I have scheduled a meeting with one of my fellow maintainers so we can discuss strategies moving forward.

Localizing and being a maintainer of Gatsby.js, so far and in the future, has brought and will bring me valuable experience. I believe that by translating the documents, I can learn this cool project. Setting up the translation has shown me how communication within the open source community is like and that there are so many people in the community that are willing to help. I will be more engaged in the project and get to know more people.

On the other hand, I also see difficulties. Making a document flow smoothly in another language while fully maintaining its meaning is not an easy task. I tried creating the README.md file and saw an immense difference between expressing an idea in English and in Vietnamese. As I am writing now, I am thinking of some words such as "framework", "component" (as in React Component) and "monorepo" for which are impossible to find a complete equivalent in Vietnamese. I guess I am going to need many help from my team.

My friend who is doing the Spanish translation suggested me to utilize Wikipedia where many words are pre-translated in a certain standard and also Google Translate but with careful review. I think they are cool tips and worth trying.

In the end, I look forward to seeing what I will pick up and struggle with in the future.

    by Vito Khang Nguyễn (noreply@blogger.com) at November 18, 2019 07:35 AM


    Ryan Marzec

    Open Source: Release 03 Contribution

    For this release in my open source class I needed to complete at least 2 pull request, one on an external project and one on an internal project that was recently started by the open source department at Seneca. For more details you can read about it here in my Release 03 intro post.

    Internal

    For the internal project I had picked out 2 issues but unfortunately I had fallen a bit behind on some work across my classes so I only completed one of the issues I had picked out, the easy one.

    The Issue
    The Pull Request

    For this issue I just needed to update eslint with the environments currently being used across the telescope project.

    External

    For my external pull request I continued on with a project I worked on previously, the issue I worked on was fairly large in size, so I decided to break it up into stages as I knew I wouldn’t have to time to complete it %100 by the release 03 due date, so my pull request is just stage one of completing and closing the issue.

    The Issue
    The Pull Request

    This is something I started in hacktoberfest , but then I only fixed a single file. After fixing the single file I learned about Butter Knife but moving forward with this in release 03 I ran into many more issues, a number of times I would make some changes that would cause the app to crash when I executed the edited section of code.

    For Activities and Fragments that did not already have Butter Knife implemented I made use to use the ViewHolder class implementation.

    Here is an example of the implementation of using a view holder class for the variables from an android view.

    public class MyAdapter extends BaseAdapter {
      @Override public View getView(int position, View view, ViewGroup parent) {
        ViewHolder holder;
        if (view != null) {
          holder = (ViewHolder) view.getTag();
        } else {
          view = inflater.inflate(R.layout.whatever, parent, false);
          holder = new ViewHolder(view);
          view.setTag(holder);
        }
    
        holder.name.setText("John Doe");
        // etc...
    
        return view;
      }
    
      static class ViewHolder {
        @BindView(R.id.title) TextView name;
        @BindView(R.id.job_title) TextView jobTitle;
    
        public ViewHolder(View view) {
          ButterKnife.bind(this, view);
        }
      }
    }
    

    As I begun this pull request there was a few things that I didn’t have time to figure out how to replace a “findViewById” variable specifically the ones called in a second view that was accessed in the current view and those within a BLANK CALL(edit this later need laptop) so I begun by using the grep command to find all instances of “= findViewById” and edited all of those files to use a @BindView variable instead.

    I think moving forward I would like to update every file to use a view holder class as recommended by the project maintainers.

    by marss64 at November 18, 2019 03:07 AM


    Ngoc Nam Khang Nguyen

    Release 0.3a: #telescope-in-motion - Is it possible to avoid coding style conflict in big project?

    Last week I mentioned project Telescope in my post about my upcoming plan for contribution to open source. It is amazing looking at it right now and see that, at this exact moment, there are 113 PRs solving 138 issues (and still counting) in the repository. I am happy to be 1 of contributors, too.

    My contribution is about integrating tools to uniform and enforce coding styles and practices to the project. This would solve a few current problems such as:
    • Indentation type and size mismatch - 1 tab vs. 4 spaces and 2 spaces vs 4 spaces
    • Newline character mismatch - Windows CRLF (\r\n) vs. Linux LF (\n)
    • Naming convention mismatch - Camel case vs. snake case (underscore-separated)
    • Anonymous function form mismatch - function() { } vs. () => { }
    • Inconsistent (or bad) code formatting in general
    • and many more...
    in the project to which there are a large number of developers contribute.

    The tools I mentioned are not completely unfamiliar in any way, they are the combination of ESLint, Prettier and EditorConfig.

    Besides that, the project is also configured to adopt Airbnb's JavaScript coding style. This greatly improves code readability and helps developers effectively avoid many common bugs. Therefore, it is favoured by many developers around the world.

    But why do we need all 3 tools?

    Ok, let's be honest, we hate adding more dependencies/plugins to our project. Then, why do we even need 3 only for code styling? I am glad that you ask!

    The answer is that each of them takes on different responsibilities (though not without overlap on each other, unfortunately), specifically:
    1. Prettier:  This tool is a powerful code formatter. It receives our code as input, generates a syntax tree and outputs the "prettified" version of the code. We can define our coding style such as 4 spaces indentation, line-ending with CRLF, each statement per line and strings must be used in single quotes... etc.
    2. ESLint: This tool is a very popular linter for JavaScript. Its job is to point out which part of the code does not align with the predefined guidelines (of Airbnb in this case) or is a bad coding practice. Also, ESLint does check for and allow configuration of code format.

      So, what differentiates ESLint tool from Prettier?

      Short answer is that Prettier worries about code appearance and ESLint worries about code quality.

      While Prettier only looks at the format of the code and nothing further, ESLint looks deeper into the semantics of the code. It can see (common) "things" that is ok but might lead to a bug later. Examples would be:

      And one of my favorite:
      In addition to the 2 tools above, we also have:
      1. EditorConfig: This tool simply overwrites the developer's editor configurations based on a config file and does not effect the code the developer writes in anyway. We need this tool although we have ESLint and Prettier because if the editor's configuration does not align with ESLint or Prettier, during the process of writing code, the linter will continuously shows error signals indicated by red underlines. Though this can be easily fixed by linter or prettifier later on with 1 click or command, it is best to avoid in the first place.
      An example of a potential behavior of any project integrated with ESLint but without EditorConfig:

      Integration Process

      Integrating the tools into project is easy. Making them working together in harmony, especially ESLint and Prettier, is a bit trickier. Thank to an article here from the professor, I am able to accomplish the task.

      After the integration, all the job of code format checking are delegated from ESLint to Prettier. ESLint only checks for compliance of code-quality-related rule. An example of ESLint config file after the task follows, you can notice that it does not contain rules about code format:
      module.exports = {
      extends: ['airbnb-base', 'prettier'],
      plugins: ['prettier'],
      env: {
      jest: true,
      browser: true
      },
      rules: {
      'prettier/prettier': ['error'],

      /**
      * Disallow Reassignment of Function Parameters
      * https://eslint.org/docs/rules/no-param-reassign
      */
      "no-param-reassign": ['error'],

      /**
      * Disallow Assignment Operators in Conditional Statements
      * https://eslint.org/docs/rules/no-cond-assign
      */
      'no-cond-assign': ['error'],

      /**
      * Disallow Unnecessary Semicolons
      * https://eslint.org/docs/rules/no-extra-semi
      */
      'no-extra-semi': ['warn'],

      /**
      * Require or disallow named function expressions
      * https://eslint.org/docs/rules/func-names
      */
      'func-names': ['off'],
      }
      };
      and example of Prettier config file which stores only rules about code format:
      {
      "printWidth": 100,
      "singleQuote": true,
      "endOfLine": "lf",
      "tabWidth": 4
      }
      The EditorConfig config file stores the list of configuration entries to be overwritten, these setups has to align with the setups above or there will be error signals during the process of writing code as an exmaple above:
      # EditorConfig is awesome: https://EditorConfig.org

      # top-most EditorConfig file
      root = true

      # Size 4 space indentation
      [*]
      indent_style = space
      indent_size = 4
      trim_trailing_whitespace = true
      end_of_line = lf
      insert_final_newline = true

      # The indent size used in the `package.json` file cannot be changed
      # https://github.com/npm/npm/pull/3180#issuecomment-16336516
      [{*.yml,*.yaml,*.json}]
      indent_style = space
      indent_size = 2
      Now... I am sitting here, waiting for all the code from my fellow contributors to come in and I will worry about the styling part.

      Conclusion

      With the combination of the trio, code in big projects, especially open surce projects, would not have any major conflict in coding style. The software would be easier to maintained and transfered to other developer team if necessary.

      Through this issue, I got to investigated an aspect in the process of creating software that I haven't looked into before. Not just with the use of the tool, thank to a lecture of the professor in class about Prettier, I also know how it is created and see the struggle without it in a project.

      by Vito Khang Nguyễn (noreply@blogger.com) at November 18, 2019 01:32 AM


      Vimal Raghubir

      Installing Darknet on Windows

      Context

      What is Darknet? For those who aren’t familiar, Darknet is an open-source framework that supports Object Detection and Image Classification tasks in the form of Convolutional Neural Networks. Darknet is mainly known for its implementation of the YOLO algorithm (You Only Look Once), which has demonstrated state of the art performance when it comes to real-time object detection.

      YOLOv3 object detection example from https://pjreddie.com/darknet/yolo/

      Chances are if you want to create real-time object detection models on your webcam or video recording, you might want to consider using the YOLOv3 algorithm that is located in this framework. That being said, onto the main purpose of this article.

      As anyone who owns a Windows PC knows, it can be a pain to set up applications and development environments. I wanted to provide some advice from my personal experience trying to install Darknet on my Windows 10 laptop.

      First and foremost, PLEASE INSTALL APPLICATIONS IN THE CORRECT ORDER. I am speaking from experience when I say that it can be a nightmare if you don’t follow this golden rule. Also, this is my recommendation for installing darknet and it may or may not work for you. Full disclaimer.

      The original GitHub repository for Darknet is here; however, we will be using AlexeyAB’s version which is an exact copy of Darknet with additional Windows support.

      Requirements

      Requirements for darknet taken from https://github.com/AlexeyAB/darknet

      I will assume that you have a GPU that has a compute compatibility version greater than 3.0. (Check if your GPU is good at this link.)

      The First Step is OpenCV

      OpenCV was a nightmare for me but hopefully, it won’t be a pain for you. I used this tutorial to get OpenCV4. I will give a brief walkthrough and some advice.

      Before downloading Visual Studio 2017, check if your PC has Visual Studio 2019 already installed. This will affect your 2017 installation since the Microsoft Visual C++ Redistributable for 2017 will not be downloaded due to the presence of more recent Redistributables on your PC. (The 2019 ones). So my advice is to uninstall Visual Studio 2019 and verify whether the 2019 Redistributables are still present in your PC’s Programs and Features. If it is then kindly uninstalling 2019’s x86 and x64 versions will fix this.

      Verify you have these after VS Studio 2017 download

      Remember to add CMake to your system PATH. If you forget to do so, you can add {LOCATION OF CMAKE FOLDER}\bin to your System Path in the Environment variables. An example CMake path is C:\Program Files\CMake\bin. If you do not know how to edit the System Path, please refer to this link.

      Likewise, if you forget to add Anaconda to your System Path, simply add {LOCATION OF ANACONDA FOLDER}\Scripts to your System Path. An example is D:\Anaconda3\Scripts.

      The code provided in the tutorial is only available from signing up for the author’s newsletters. So when you see the popup below, please click “Download Code” and sign up. You don’t pay anything and you can honestly unsubscribe after you get what you need. I don’t know the author nor am I trying to promote his newsletter, however, it is a massive timesaver.

      Access to the installation scripts in the tutorial

      I tried for 2 days with a different installation guide so trust me when I say it’s a lifesaver. Once you tested that OpenCV4 is working, congratulations you passed the hardest part of the installation!

      The Second Step is CUDA

      I will make your life very simple. Click this link and select Download. Of course, if you have Windows 7 or 8 then change the Version.

      CUDA 10.0 Download for Windows 10

      Once you run the installer, just keep clicking Next and verify that you do not encounter the screen below while installing.

      Your Visual Studio 2017 did not install properly

      If it did then you need to uninstall Visual Studio 2017 and redownload. If you did not encounter the message above and CUDA was successfully installed then you can move on to the next part!

      The Third Step is getting CuDNN

      To download CuDNN please click this link. You will need to register for an Nvidia Developer account before getting CuDNN. Once you register, agree to the terms and conditions and click the installer as shown in the screenshot below.

      Once the file has been downloaded, extract the contents directly to your C drive. Once you have finished extracting, verify that the Cuda folder exists in your C drive as shown below.

      Checking for Cuda folder presence

      After that open up your environment variables, and add C:\cuda\bin as a new entry to your System Path. Congratulations you have installed the major requirements!!

      The Fourth Step is getting vcpkg

      The link to get the vcpkg library manager is here.

      You can download the zip file for vcpkg as shown below.

      Downloading vcpkg zip file

      Once downloaded, extract the contents to a location of your choice. After that, open up a Command Prompt and navigate to the vcpkg folder. Once in, run the command:

      bootstrap-vcpkg.bat

      After this bat file is finished its execution, run the command below:

      vcpkg integrate install

      After running these commands successfully, please open your Environment variables and Click “New” in the System Variables section.

      New system variable

      Name the new variable, VCPKG_ROOT and make it point the location of the vcpkg folder.

      Also, define another system variable named VCPKG_DEFAULT_TRIPLET and set its value to x64-windows.

      Open a Powershell window and type the commands below to prepare vcpkg for the darknet installation.

      PS \>                  cd $env:VCPKG_ROOT
      PS Code\vcpkg> .\vcpkg install pthreads opencv[ffmpeg]

      Final Step is getting and installing Darknet

      The moment we all were waiting for is finally here! Navigate to this link for the darknet code. Once you download and extract it to a location of your choice, open a Powershell window in Administrator mode.

      Navigate to the location of the darknet folder and before you can build it, you must ensure that you can execute Powershell scripts.

      Run the command below to verify your execution permission level.

      Get-ExecutionPolicy

      If this returns unrestricted then you are good to run the final command, however, if it is not then run the command below.

      Set-ExecutionPolicy -ExecutionPolicy Unrestricted

      Select “A” or “Y” and then run the final command.

      .\build.ps1

      Once this finishes executing then your darknet repository should be installed. Congratulations!

      Don’t forget to run the command below and type “A”.

      Set-ExecutionPolicy -ExecutionPolicy Restricted

      To test darknet, simply running the command below should yield some output without any errors.

      darknet.exe detector test cfg/coco.data cfg/yolov3.cfg yolov3.weights -thresh 0.25

      YOU DID IT!

      Once again this isn’t a one size fits all solution, however, if it does work for you then I am very happy that I could help. If it doesn’t then please refer to this link to try installing darknet with Cmake-GUI instead.

      To see the various possibilities for training available to this darknet framework, please refer to this link.

      Thank you for reading all the way up to this point and hope this guide helped you to get darknet installed on your PC. Take care for now!


      Installing Darknet on Windows was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

      by Vimal Raghubir at November 18, 2019 12:36 AM

      November 17, 2019


      Andre Rosa

      The CDOT Telescope Project 03 and External PR

      Internal PR

      Last week I finished the fix coding with a function and a JEST test, However I ran into some problems before my PR (issue #74) was accepted:

      1) The test function tested in the same scope several URLs for validation. This is a bad practice. The correct way to create tests is to develop atomic function that test only one situation. Therefore, I had to update my tests and exploded it in several smaller functions.

      2) I faced some conflicts because of the package.json file as some other students were also updating and making merges to the repository. Finally, professor David helped me with the rebasing. The basic sequence of commands as follows:

      git checkout master
      git pull upstream master (to sync local with upstream)
      git checkout issue-xx (to return to the issue branch)
      git rebase master -i (for interactive rebasing)
      git push origin issue-xx -f (to force the push without other changes)

      An interesting tool I started to use was GitLens, this VS code extension shows in code the authors for each change in the file. A good way to find out if it was you who added that small change in code by accident.

      As I am still not very secure on rebasing, I went after a tutorial that could help me. There is a good article on rebasing in Bitbucket page and explains the use of interactive method as well the ‘squash’ and ‘fixup’ commands: https://www.atlassian.com/git/tutorials/rewriting-history/git-rebase

      External PR

      For the external PR I continued to work with a character generator for games. I choose this project because the maintainer was nice and helpful during my first commits.

      The job was to create a method to calculate the armor of the character based in some traits and equipment. I had to do some research to find out the sequence of operations to get to the final armor. After some tries and tests I managed to get to a valid solution, I also created a JEST test for the new feature, but this was not added to the final PR because the project did not added JEST yet.

      The issue is here and the PR is here.

      by André Rosa at November 17, 2019 09:44 PM


      Haichuan He

      OSD600 Release 0.3 Pandas

      The external project I chose is Python pandas library. Python is the language I am most interested in getting good at and I am interested in mathematics and data analysis so contributing to pandas will be very exciting for me.

      The issue I found is about incorrect data shift result when columns have mixed types for a DataFrame. The link is as below:

      DataFrame shift along columns not respecting position when dtypes are mixed · Issue #29417 · pandas-dev/pandas

      At the beginning I thought this might be easy to fix as it probably related to how data type of each column is tracked and determined after shift function is being called. I spent quite a lot of time setting up the environment and reading through the code base to figure out how it is organized. However, after some investigation this issue is way complicated than I thought.

      The DataFrame class is inherited from a Generic object class, which also has a shift function. The DataFrame is treated by pandas as blocks of data with a particular type. When mixed type data is used, the data is broken down into separated blocks with different types using index slicing. Below is an example:

      df = pd.DataFrame(dict(
      A=[1, 2], B=[3., 4.], C=['X', 'Y'],
      D=[5., 6.], E=[7, 8], F=['W', 'Z']
      ))
      df
      A B C D E F
      0 1 3.0 X 5.0 7 W
      1 2 4.0 Y 6.0 8 Z

      The original data have mixed float, int and letter values. In pandas, non numerical and non boolean values are treated as object. The DataFrame itself is also treated as object.

      df.dtypes
      A      int64
      B    float64
      C     object
      D    float64
      E      int64
      F     object
      dtype: object

      When shift function is called, the sequence called is as below:

      shift in frame.py, shift in generic.py, shift in manager.py, shift in block.py

      Using print function to identify key variables to display, I found how the blocks are determined:

      BlockManager
      Items: Index(['A', 'B', 'C', 'D', 'E', 'F'], dtype='object')
      Axis 1: RangeIndex(start=0, stop=2, step=1)
      FloatBlock: slice(1, 5, 2), 2 x 2, dtype: float64
      IntBlock: slice(0, 8, 4), 2 x 2, dtype: int64
      ObjectBlock: slice(2, 8, 3), 2 x 2, dtype: object

      When shift operation is performed, it passes each block to the manager shift function as below:

      def shift(self, **kwargs):
          print("in manager")
          return self.apply("shift", **kwargs)

      This algorithm caused three column changed to NaN whereas only the first column should be NaN.

      FloatBlock:
      [[3. 4.]
      [5. 6.]]

      after shift:
      [[nan nan]
      [ 3. 4.]]

      IntBlock:
      [[1 2]
      [7 8]]

      after shift:
      [[nan nan]
      [ 1. 2.]]

      ObjectBlock:
      [['X' 'Y']
      ['W' 'Z']]

      after shift:
      [[nan nan]
      ['X' 'Y']]
      Result:
      A B C D E F
      0 NaN NaN  NaN  3.0  1.0  X
      1 NaN NaN  NaN  4.0  2.0  Y

      I tried to cast the original DataFrame to all object by using:

      self = self.astype(object)
      result:
           A  B  C  D  E  F
      0  NaN  1  3  X  5  7
      1  NaN  2  4  Y  6  8

      The result seems to be correct except that the float type is not maintained, which causes test to fail, so I cannot submit a pull request based on this solution.

      I need to further investigate the code and see where I can implement casting to cast back the types based on which type each type supposed to be.

      by Tony He at November 17, 2019 08:21 PM


      Sukhbeer Singh Dhillon

      Stage 1(Contd): Targeting code to optimize

      In my previous blog, I began exploring the xz utility. From my analysis of the perf report, I found that the bt_find_function is being called maximum times, almost 40%. Inside the function itself, the hotspot seems to be a cmp instruction which is checking if a register is 0.

      Perf annotation for bt_find_func on Aarchie

      My first idea is to try and use the bitwise xor instead of the subtraction.

      I had contacted the main dev for this project Lasse Collin regarding ideas on what to optimize. He gave me the suggestion to look into the code in the memcmplen file. It is optimized for x86 to work for unaligned 8-byte access. Other architectures use 4-byte access. He advised looking into this function which will have a potential impact on compression.

      The file mentioned above provides a function that compares two given buffers and returns the number of bytes that match using the uint32_t datatype. The number returned is always between the number of bytes already compared and matching to the limit up to which to compare the buffers.

      A first glance into the code suggests that intrinsics are being used to carry out all sorts of instructions like load, store, add etc. My goal will be to create an elif directive that will be true if it is an ARM platform. I will be following the below mentioned guides to understand the given code in x86 and the equivalent or improved ones for AArch64 SIMD.

      So there you have it, my plan. I have two ideas basically. One seems a bit silly but is based on my beginner’s knowledge about processors and assembly code. This is the result of my benchmarking and profiling results. The other one is to do what the person who knows this code inside out suggested.

      by Sukhbeer Dhillon at November 17, 2019 06:53 PM


      Minyi Chen

      OSD600 3.0 Telescopen update

      As I mention in the previous blog, we are going to contribute to telescope. 
      Click here to see the GitHub.
      I checked the issues that we have so far and cannot find any issues that I can work on.
      So I discussed with my classmate and create the issues for us.
      After our discussion, We decided to make a function to count how many blog come from different domain.

      Issue -- Implement blog post count by domain
      Click here to check the pull request.

      Issue -- Test the function of counting each domain post.
      Click here to check the pull request.

      Issue -- Updarte blogDomainCounter to pass tester
      Click here to check the pull request.

      When we discussed to make the function to count it, we though it should be done soon cause it was not so difficult. However, when we actually open a file and started working on it. we met so many problem such as:
      1. How we can get the urls.
      2. How to get the domain.
      I read all the code we had so far and understand what other people  did so far. Finally, we figure how to do it and finish. I was very happy to create the pull request. However, after some reviewer reviewed the code, there are so many problem came out.
      I re-write m code to meet their requirement.
      After I fixed it and push my commit again. There is some problem in rebase.

      Here is someone teach me how to solve the problem:

      External Issue:
      I also did an external issue in the week.
      This project is using React to do it. When I found this issue, I though it was an easy issue to fix.
      However, when I stepped into it. It was harder that what I though.
      It used yarn. Then I need to install it and learn it how to use it to run this project.
      It was not easy to find out where the problem came from.
      This project created functions in scss that I have never did it before.
      I was so surprised that we can do like it.
      Finally, this pull request had been merged.
      Click here to check the issue.
      And Click here for the pull request:



      Conclusion:
      Even though I need to spend a lot of time to fix an issue, I felt so happy cause I can learn a lot during I run the project, find the problem and fix it. Communication is very important when we collaborate with people. Sometime we did something we though it was correct, however we used the wrong way. Never assume something when we create a function.





      Previous: OSD600 Release 3.0 Telescope 

      by Karla (noreply@blogger.com) at November 17, 2019 06:03 PM


      Haichuan He

      OSD600 Release 0.3 Telescope

      For release 0.3, I work closely with a classmate to develop a function to count the number of blog posts from different domains and the test for it. The issues are listed below:

      Approach

      This is the first time we work on a project working with blog feeds. We took some time to investigate the code we had so far and found that someone had already written a function to get all blog feed urls from the website. So we decided to use the same format of this function output as input to our function, which is an array of json objects with one property “url”.

      We wanted to get the result that has the domain names and number of times it appears in the feed urls. We decided the output to be an array of json objects like below:

      [
        { name: 'wordpress.com', count: 2 },
        { name: 'medium.com', count: 1 },
        { name: 'blogspot.com', count: 7 },
      ]

      Soon we realized that we need a function to get the domain name based on url, so we used url.parse function to get the hostname and remove the user name portion so that only the domain name is remained. The core function loops through all feed url objects, parse to get domain names. For the first object, it automatically pushes into an array with count equals 1. For the remaining objects, it first testes if the same domain name already exists in the output array. If yes, add count by 1. If not, push a new domain object to the array. The function finally returns the domianSummary array that can be used for further analysis.

      For test, we simply created a sample input and expected output, and use standard expect function to check if result matches the expected result.

      test('Count domains from blog posts', () => {
      expect(countBlogDomain.blogDomainCounter(feedUrls)).toEqual(domainCountResult);
      });

      Pull Requests

      The pull requests are listed below and they both got merged. Further improvement may be needed.

      Lessons Learned

      This is my first experience working on an open source project with a group of people using standard processes. To get our code merged took a lot more effort than I expected. First, the code needs to pass automated testing for formating which force us to follow eslint format of javascript code. Second, the code needs to be easy to understand for future potential maintenance which means it needs to be properly commented, minimizing duplication of code, as well as following other best practices. For example, initially we used for loop with index variables instead of foreach loop for array object. The code reviewer asked us to change to use foreach loop as it is easier to understand. Last, working with many people simultaneously requires strong git skills. The upstream master branch is always changing, we need to pull the recent update and rebase our own branches before submitting pull request.

      Overall, it is a valuable experience to work on a project that is similar to real work. It made me think to put more effort to improve code quality including formatting, how the code is organized as well as making statements more understandable. Looking forward to working on more issues for the telescope project in release 0.4.

      by Tony He at November 17, 2019 04:10 PM


      Manan Patel

      The Big Disaster

      Wow, release three, WHAT A NIGHTMARE. For release three of DPS909 course, we had to submit a minimum of two pull requests; one internal, ran by Seneca, on the project telescope and the second one external any other project that's not the telescope project. For the internal project, telescope, I was working on an issue to set Travis CI up so it runs on Mac, Windows, and Linux and send in a pull request. 
           
           When I was working on travis.yml, file name, I was having a lot of difficulties because I have never worked with Travis before. The good this was Travis had pretty decent documentation so I could set up the Redis server and homebrew as well. Once that was done I tasted my code and to do that I had to make a pull request because when I tried to test it locally it didn't test the travis.yml file since I didn't have my system set up for Travis. So when I looked into how to set it up locally I found a StackOverflow entry explaining all the sets. So I followed them and found out my mac is full, I don't have enough space to set up the docker and Travis so I started deleting unnecessary files and apps that I didn't need and used any more. That gave me four gigabytes of space, not enough, but just enough for me to set Travis and Docker up locally. So I tried setting it up and failed miserably. So I researched a bit more and I started noticing some issues with the performance of my mac because there was not enough space. Then I deleted docker and uninstalled Travis. Following all that I decided to set in a pull request and let the GitHub Travis test my code. Great I got that started and every time I git added, committed and pushed my change to GitHub, it ran Travis. Now I started reading up on Travis syntax, how to configure on travis.yml, looked at other example code on Github for travis.yml and looked on StackOverflow. From all that searching I learned how to add addons for mac, essentially setting up the VM for the Redis server. Then how to run the Redis server on mac with an if statement so when the test runs for windows it doesn't run the server start command that mac would run. Another thing I found out was that services are not supported in mac but are in Linux so when the yml file had the syntax "service: redis-server" it started the server for Linux and that's how I came to find out the server needed to be started manually. After I started them manually with the following commands: "redis-server /usr/local/etc/redis.conf" for mac and "server.exe" for windows. When I did that during the Travis test the servers would start and be in a state to accept a connection but never received the connection. That puzzled me so I asked for help and I found out it would be started during the test so I just needed to install them or so I thought. Following what I thought I understood I removed the server start commands and ran the Travis test again. This time it came out as there is a problem with the asynchronous test function so I thought great I think I set up the system properly because I didn't work on test functions. But turned out during the test the Redis server had not set a default URL. So that makes all the following tests fail. In setting up Travis ci for windows I had one major problem but all the other steps were the same, setting OS as windows, installing Redis for windows and starting it. The major problem was the prettier format. For windows EOL, end of the line, the format was CRLF but Travis needed just LF like how mac and Linux have their EOL ad LF. I had no idea how to set this up. So I searched a bit and found out that this can be turned on and off in git config global core.autocrlf. Then I inserted that line as a "before install" command in and ran Travis. It did nothing. So I searched a bit more and found a code block on StackOverflow to fix the exact same problem I was having. So I took that code block and used it in my yml file and I commented the link right before the code block. Finally, I pushed my code and it got reviewed and merged. Then my prof sets up a new issue to fix why the Travi ci test was failing for all other commits. This scared me a little because did I break the whole system. Turned out I did with my merge. Then he recorded the whole file to fix the fix I should have fixed. Later I found out that the problem was caused because the Redis server was not starting since it didn't a default URL.

           This was a really big mistake I caused and I realized it after. The lesson learned is never messed with setting up the configuration of a system ever again!!!! Nope, I am just joking but what I actually learned was 1.) if a test fails before merging on GitHub you are doing something wrong, take a second, third or even a fourth look and find out why it's not working properly. If you still can't ask for more help and clarification and do a deeper dive into searching. 2.) never push code that fails CI because it will cause huge problems for other developers who are depending on it. 3.) if part of the tests or CI is failing, move that bit to a new pull request against landing everything at once.

           For the external after my confidence in myself failed, I tried to do something simple. So I found a Microsoft project on GitHub called fast-dna which is about "an unopinionated system of components, development tools, and utilities used à la carte or as a suite to build enterprise-grade websites and applications"(fast-dna, line-1). This project is all typescript. Honestly, I choose this project because there is a simple issue I can work on here when the scroll bar should show a number when changed and it was a good way to refresh my memory of typescript. But the second objective failed I couldn't remember anything I learned of typescript. So on the same project, I found another issue of moving a file from once component package to another component. I did that and sent in a pull request but it failed their circle ci test. So now I gotta go back and look at where I am failing. So learning from my previous mistake I have not merged this code yet but I will send in for another review once I solve this problem.

           This release, 3, has been a great learning experience for me in why testing is important and not merging anything that doesn't pass the test. Even with the Microsoft project, I failed the build dependencies so I think it might be due with not changing the path of the file import in some component but I will look at it again. This is one lesson I will never forget moving forward.

      by Manan Patel (noreply@blogger.com) at November 17, 2019 04:36 AM


      Ana Garcia

      *Insert Git Pun title here*

      Gatsby-es Links: Repository| Issue | Pull Request Working on a project for more than a couple of days was a fairly positive experience. Though, I will say editing a large document never is. My experience translating was different this time — more difficult and time … Continue reading *Insert Git Pun title here*

      by semortea at November 17, 2019 01:20 AM


      Neil Ong

      Release 0.3: Part 2

      Hello friends,

      The weekend has come and that means my release 0.3 was due and has been finished. As I told you guys last week that for release 0.3 we needed to have both an internal PR and an external PR. Where our internal PR had to be merged for it to count.

      Internal PR

      For my internal PR I decided to create a test for a feature that was going to be landed, which was the ability for blog planet called “telescope” to be able to send out emails to people if something went wrong. Since someone else was implementing that feature I worked on helping review his PR and ensuring that it was correct. If you follow the link, you can see that we had multiple back and forths trying to get the code to land. Where I requested a few changes before I approved his changes. These changes were essential to help myself create a proper tester for his code. I requested that he change his function to resolve a promise by returning the recipients that received the email. This allowed me to check the output much easier.

      Now, when I started to write my test I messed up with the way I rebased my branch onto my master. Instead of checkout to my branch that I was working on. I believed on my branch and tried to rebase my master onto my branch. I don’t remember exactly the sequence of commands I ran, but I know I messed up and that I have to be more mindful of that going on. This caused my git history to have many “merge branch ‘master'” commits. After learning that I had messed up, I asked a friend to help me with rebasing and squashing and fixing some of my commits, but we couldn’t fix everything unfortunately.

      When I created the PR I also had some back and forth with the person who wrote the code I was testing. We tried to get the test to be better on how it gets the credentials it uses test and we ended up figuring out how we wanted to do it. One thing we did find though was that the original function was failing the test since it was using and old format. So, it was decided on the internal slack that we should .skip() the test until it was fixed. We did end up fixing the issue with the original code and the test failed for the right reason since we didn’t have the connection to our server. After this, we put the test back to be a skip and the changes were approved and finally landed.

      External PR

      So, last week I told you guys that I was working on a fix for an issue for ODK Collect. While I am still actively working on that issue, I found out it was actually a lot more work and the bug was harder to figure than I had thought. This is because the codebase was pretty complex and large and it was written in android, which is something I’m not as familiar with. I still plan to work on and try and figure out the issue, but for this PR I decided to work on something more attainable. I found a new issue for meshery, which is tool to benchmark the performance of service meshes. The issue was about changing the text of a certain button on the Mesh Service page. To fix this issue, I went into the codebase looking for where the value of that button was being set. Once I found it on vscode, I set it to the desired text. I feel like this issue wasn’t was big or as hard as I wanted it to be, but I did learn how to try and look more effectively through a large codebase, which I hope will help with the ODK Collect issue.

      In the End

      I learned a lot about working on an open source project. The speed and the chaos at times can be crazy. Having code being constantly change and code landing that can create issues for other individuals. It really opened my eyes to a different side of open source development. It leaves me wondering how on earth big companies and projects maintain their git properly.

      by Neil at November 17, 2019 12:57 AM

      November 16, 2019


      Lexis Holman

      Another Project Update

      Another quick update; but first, for those of you who may not be totally familiar with my attempt at optimizing an open source project can check out the previous posts here:

                      https://penguins879376500.wordpress.com/category/school-work/project/

      The reason I needed to post again today was because last night was somewhat productive; or well not really productive, but I did a lot of work anyways and you get to hear about it.

      So last night was when I started really digging into the source code of the project and even attempted to make some modifications. If you look back on this post:

                      https://penguins879376500.wordpress.com/2019/11/15/project-update/

      You will see that I was attempting to modify some of the loops within matcher-bm.c within libclamav. I thought that it would be possible to potentially modify some of the loops in a way that more than one character could be checked at each iteration. The issue with this thinking was that I was not aware of the fact that this function is in fact a computer science algorithm called a “boyer-moore string search algorithm.” For more info you can checkout this link:

      https://www-igm.univ-mlv.fr/~lecroq/string/node14.html

      Where mentioned in this link that, “The Boyer-Moore algorithm is considered as the most efficient string-matching algorithm”.

      I (think I) was right about one thing, this function is doing the heavy lifting in terms of comparing character and string values. However the above mentioned search algorithm with c style linked lists used individual nodes to contain the patterns and offsets that are required for the algorithm to work. This meant that it would have been required to get the next node for comparison. For our attempt at optimization, we needed to get the address of the next data structure from the next pointer within the gotten node and access the individual element.

      Attempting this in practice only slowed the software down, the extra loads and compares from memory only got in the way of how the algorithm was supposed to search, though it did not stop the program from making the correct selections.

      The parts of the program that where showing up as busy instructions were just processing and incredible amount of information simply by looking at a single character. Which is exactly why we had the movzbl instruction at almost 30% of the total function run time; that specific instruction was responsible for loading the character for comparison. Wouldn’t it be great if we could load 2, 4 or maybe 8 characters for comparison at a time? Of course, and I am still hopeful to be able to do that before the time in this class runs out; however it must be done in a way that does not break the algorithm.

      After playing with this block of code until about 3am, I was able to find that with adding debug printf statements to individual sections of interest that this program is unique (to me) in how it is able to compare a massive amount of data in a very short amount of time. That is by checking individual characters it is able to process and compare almost 8GB of data in about 2 and a half minutes.

      This is a great example of how a community of people can come together to make something about as good as it can get. Whatever happened with this project, it has defiantly increased my appreciation for open source technologies. Not that I ever had a doubt in the “bazaar” style of development.

      Now I must finally mention that, I am not completely giving up hope on this project yet, there is still one little glimmer of hope in the form of a character comparison loop within matcher-bm.c at line 345 (as of this writing) that looks like this:

        found = 1;
        for(j = 0; j < p->length + p->prefix_length && off < length; j++, off++) {
           if (bp[j] != pt[j]) {
                  found = 0;
                  break;
            }
        }

      This small loop of about 7 lines of code, does not seem to run particularly busy. It is however one of the last checks in the control of its parent loop. If the character bp and pt at index j do not match, then the break statement ends the loop and execution jumps to the starting for loop at line 276. This is where p, which is a type cli_bm_patt linked list pointer; is then advanced to get the next pattern structure and searching continues.

      Unfortunately as each node in the linked list must traverse the above part of the parent loop to see if it is appropriate to be used for a match attempting to work on multiple nodes is not going to work. This is not an optimization that can be handled with vectorization and instead needs to be tackled with parallelization; a task that is already handled by clamdscan.

      My last shred of hope for any kind of optimization within this block of code is that we can somehow try and make this loop take advantage of vector registers and compare multiple characters per iteration. To do this within the X86 environment we MAY have a couple instructions available to us, they are:

      MPSADBW – Compute Multiple Packed Sums of Absolute Difference

      https://www.felixcloutier.com/x86/mpsadbw

      &

      PCMPESTRM – Packed Compare Explicit Length Strings, Return Mask

         https://www.felixcloutier.com/x86/pcmpestrm

      From what I understand string compares can be difficult to use in terms of vectorization. This is due to that fact that we could potentially load values past the end of the array if we are not properly aligned, or we are comparing strings of different sizes. But by using these instructions, the first of which showing more promise we MAY be able to get a tiny bit more performance out of this function.

      Before I get anyone’s hopes up, I need to note that the way this must be done will be architecture specific. The best way I can think of is with intrinsics which will require we use some kind of pre-processor logic. My overall worry is this may overly complicate this block of code to the point it may not be worth the minimal performance gain that will be achieved; let’s try anyway.

       

      Cheers,

      ElliePenguins

      by elliepenguins at November 16, 2019 11:55 PM


      James Inkster

      git pull a day keeps the conflicts away

      Working in open source, can be beautiful and destructive. It’s an interesting ecosystem of different individuals all around the world, contributing to various different projects they believe in.

      These last couple of weeks felt like I had just entered into an old 1960 western starring Clint Eastwood. I am including Eastwood because of the amount of squinting I had to do for the long periods of time I was looking at a monitor.

      Open Source Thoughts

      There is a lot to take in – I worked on two pull requests, one for an internal project that was moving the speed of light, and the other Visual Studio Code, something that I use and appreciate very much, and feel is a strong IDE. I don’t believe in making a blog solely on complaining, so when I talk about my responses below. These are things I felt were very beneficial for my growth.

      Things I Liked
      • Any project
      • Friendly open source community
      • Reading tons of different styles of code
      • Witnessing how large a project can be
      • Discovering new tools and libraries relatively simple
      • Learning git to a higher degree.
      • Different coding backgrounds coming from all over the place
      Things I Disliked
      • Waiting for previous issues that I could do myself to be done before I could proceed
      • Conflicts, and people cheating the system.
      • You can just as easily join and leave a community
      • Hard to understand the overall work flow of the project
      • Hard to communicate in real-time on projects
      • Individuals who aren’t owning the repository making executive decisions (in terms of coding style.)

      Well, then…

      Keep in mind these were spent over a short period of time, and just the problems I saw for the probably 20 hours I put into working on both my internal and external request.
      I don’t feel like either the pros or cons will go away. The exception being when I worked on visual studio code. Someone from microsoft has to approve or disapprove your pull request and issue. However, many of the projects don’t have that level of service. It’s actually odd that microsoft does, huge corporation that has billions of dollars utilizing on an open source community to help push them forward is a really weird conundrum, and I’m assuming it’s probably to not stifle the level of creativity that can be input into their product that they look at as the best approach to stay ahead of the curve.

      What were the pull requests?

      You can see the two ones I did here and here.

      Visual Studio Code


      The first one, Visual Studio Code, is adding the Cntl-shift-C feature and Cntl-shift-V feature to Windows terminals. Now, who uses it? Well, apparently it’s becoming standardized in many terminals, and I believe already the standard way of practicing in Linux. When an individual has to switch back and forth through systems, thats where the issue lies. Muscle memory fails them.

      Now, I initially got stuck because the issue read cntl-shift + *. You can see the issue here.

      So I scoured the internet, far and while for this mysterious combination of cntl-shift and *. Yes, I couldn’t see a tree from the forest…or a forest from the tree. I’m not sure the saying. The * representing an infinity possibilities. It wasn’t a specific function attached to ‘*’.

      Now, eventually I tried somethings, and my code was actually incorrect, however due to a custom setting I had on my computer, it did not work in the submitted copy, so I had to make some changes.

      I found working with visual studio code, very simple, easy, super helpful staff, and I hope to continue to work with a project that has a very nice working structure. Great use of maintainers, little hassle for the merge, and very clear cut coding guidelines.
      I enjoyed this despite some of my wasted time setting everything up quite a bit.

      Telescope

      This project is far more interesting, created far more issues, and made me re-think about opening or starting a project on open source. And to clarify, open source can be a really good thing.

      Well, because this was a brand new project, I felt there was a lot of growing pains that the open source community isn’t as equipped for. A lot of problems are stumbled through or on during the start of a new project. Now, these aren’t problems that can’t be fixed, especially with the amount of people working on this however, depending on where you are at, and when you want to contribute thats where this will let you down. For my particular times issues exploded, like a volcanoe.

      Everyone trying to jump on the issues that are easiest, ones with large amounts of code easily added or ones they felt comfortable doing. The issue? Well, a good example was the “express server.” This was something that I believe was done, almost a week later after the issue, and done extremely well. However, the parts that I wanted to interact with did not need that same level of quality. I needed a rough express server, just so we could move forward faster with SAML introduction. This was an issue that directly affected me, and depending on how long or the other individuals time, would dictate what I could have done. However to deploy my own express server, would have taken me less time, and easier to test. Frustrating experience as I don’t necessarily think in open source that doing more is always better. It’s staying within your issue and accomplishing your issue I feel is more important. The faster that issue is dealt with, creating your next issue should be relatively simple, this allows for a much simpler flow.

      There’s also the case that I saw others run in, which is styles of code. Well, early contributors got to kind of “dictate” this, or individuals who chose the large amounts of code. This is an odd scenario to have happen, and one that only is kind of near the start of the code when so much code is flying around. I almost feel like its better to have some ground work set-up before an open source project is opened to the public for this reason, as I think there are some questions that need to be answered prior too the start of a project to lessen the growing pains to a ton of devs that are relatively new. I do envy the next class that gets to work on this project. As a lot of these things will be dealt with by the time they are up to perform.

      They’ll also not have to hopefully deal with the crazed package-lock.json file, and its ever changing ways because hopefully by the end all the packages we need will be installed. Although thats an issue, I don’t think it’s one that will pop-up on the regular due to the vast amount already added.

      To say these weren’t great character building scenarios would be dishonest, they definitely helped me think more critically on how I would like to manage my own future projects. And for that I am greatful, while someone else may jump into a similar situation, perhaps if we didn’t have the great David Humpries at the helm who was more knowledgeable, I truly do feel the ship could just as easily sunk with someone with less experience.

      Limits

      My last post talked about my limits, and I do feel like I was able to push through these issues, however much like Dragon Ball Super ended, the truth is, you push through these limits through the help of friends.
      https://www.youtube.com/watch?v=g2zMzspzbEM
      For a scene that accurately reflects my feelings which are that Jiren (The bald headed super muscle-y dude.) is the problems we face, and goku is the embodiment of pushing forward and the impact friends can have on us.

      by grommers at November 16, 2019 06:36 PM


      Bowei Yao

      Open source and Windows

      Open source and Windows… they just don’t seem to get a long. In the short amount of time I had since my last post, I did a few things:

      • uninstalled docker from windows
      • installed ubuntu 18.04 (bionic) dual boot on both my desktop/laptop
      • ubuntu 18.04 (bionic) doesn’t come with the library make pre-installed! I was shocked…
      • my desktop doesn’t have a built-in wifi adapter, so I had to tether my phone’s hotspot in order to install the driver for my usb wifi adapter first, which namely is done through sudo apt-get install rtl8812au-dkms, which installed libraries like build-essential, make and all that good stuff
      • installed yarn. Funny thing about yarn is that if you don’t register the appropriate repository for it, it will automatically redirect you to download some package named cmdtest, which is not what you want. See this link: https://yarnpkg.com/lang/en/docs/install/#debian-stable
      • installed nodejs. If I just installed nodejs from the official supported ubuntu packages, it will just install version 8 for me. Version 8 is not sufficient for most of the repos I’m working on at the moment, they all require 10+. See this link to install later versions, such as 10, 12, and 13: https://tecadmin.net/install-latest-nodejs-npm-on-ubuntu/
      • installed miniconda for environment management
      • in my newly created conda environment, installed pip, pipenv, ffmpeg, invoke, gettext, docker-compose
      • installed docker. Again, need to register their official repository. If you just do sudo apt-get install docker it will give you something else. Follow the instructions here: https://docs.docker.com/install/linux/docker-ce/ubuntu/
      • assign appropriate privilege for my user to use docker without root. This was particularly important for me due to the setup scripts being dependent on them: sudo usermod -aG docker your-user
      • installed mysql, sudo apt-get install mysql-server
      • created new sets of ssh keys and added the public ones to my github account ssh-keygen -t rsa -C "bwyao@myseneca.ca"

      All these seem like a lot of work, and they are indeed. But I’m pretty happy when the projects finally get spun up properly locally:

      The foundation.mozilla.org page, on my local machine: (https://github.com/ragnarokatz/foundation.mozilla.org)

      The voiceweb homepage: (https://github.com/ragnarokatz/voice-web/blob/master/CONTRIBUTING.md)

      However, I’m still having trouble getting this one to setup properly: https://github.com/mozilla/addons-code-manager

      After spending hours trying to get it setup despite the simple instructions in the readme, I’ve created a ticket asking for help: https://github.com/mozilla/addons-code-manager/issues/1246. Once this is resolved, I can proceed to work on ticket: https://github.com/mozilla/addons-code-manager/issues/1193. This one utilizies a package named stmux for running multiple processes in one terminal:

      I can’t seem to get the last process yarn server-dev working.

      by Schrodinger's Cat at November 16, 2019 05:39 PM


      Timothy Morris

      DPS909 Release 0.3: External and Internal projects Part 2

      Last week, I talked about my next assignment in my open-source class which is to participate in the internal project while also finding another issue from an external project to work on. I mentioned that for the internal project, I would create a code of conduct and I also mentioned that I found an external project issue to work on. After completing both tasks, I’ve learned a few things along the way.

      The issue I came up with for the internal project was pretty simple, creating a code of conduct for the telescope repository. It may not seem like a big contribution to the project because it’s just a small document that doesn’t have to do with coding but that’s not really true. A lot of big GitHub projects have a code of conduct to display the type of behavior that the creator of the repository wishes to see from all of the contributors. Having a code of conduct shows that this is a serious and friendly environment for everyone and anyone that doesn’t follow the rules will be disciplined. My original idea was to create one from scratch by myself. I did some research on code of conducts from other projects and tried to come up with something but I realized that they all pretty much say the same thing. I also tried to think about what we would need specifically for this project but I felt the need to ask around. I was going to ask my prof and a few other students in class the next day but while I was working on my external project issue, I noticed that the code of conduct in that project was sourced from contributor-covenant.org. I decided to investigate and found out that this code of conduct has been used by a variety of other open-source projects as well. I decided to go with this one and submitted my pull request. I was afraid that people wouldn’t like the idea of me using a template instead of making one from scratch but it turned out they liked it after all. They approved my request and merged it into the master branch. Contributing to an internal project seemed to be stressful at first but it turned out to be fine based on this experience. Once all of that was complete, I got to work on my external project issue.

      For the external project, my prof wanted to see us take on an issue that’s a bit bigger than all of the issues I did during hacktoberfest. Throughout my hacktoberfest adventure, I added a link to a website, simplified navigation for a website, fixed a bug in a react app, and displayed a readme on a web page. These issues may have been small but it did display growth so I needed to find something a bit bigger than my last issue (displaying the readme). My prof mentioned that I could take an issue from a project I previously worked on before and with that in mind, I remembered seeing an issue to design a quiz game mode for the emojibops site. Initially, it didn’t seem like a big jump from my previous issues but this was implementing a simple game mode on a whole new page on a website. I personally think it’s best I take little leaps in progress instead of just jumping way too far from my previous spot. I also recall having a friend of mine do a similar issue last month and I was originally planning to ask for some advice if he was willing to offer it but I decided to try things out myself before asking for help.

      Creating the quiz page started off pretty simple since it was mainly copying and pasting certain logic. Once I finished with the copying and pasting, I had to make a small edit to a function that would generate the array of emoji cards for the quiz. Once I fixed up the generation, I had to think about how to make the game and of course, I wanted to go the easier path to save time. Displaying a set amount of cards on the page at once and having the user check for the answer after seemed to be easier than just displaying one card at a time so I went with that method. Next, I had to come up with a way to display the text boxes under the cards and check the answers. This part of the coding process was pretty interesting to me cause I came up with two ways of displaying text boxes and checking answers. I tried to do a regular text box from HTML forms and I tried a <p> contenteditable tag that I kinda copied from my previous note taking app. Eventually, I decided to go with the HTML form text box instead of the contenteditable tag because it just seemed to look better in my opinion. For checking answers, I decided it would be cool to add a checkmark emoji next to correct answers and a red X emoji next to incorrect answers. I did this to fit with the whole emoji theme and I’m sure the owner of the repository would like this little touch a lot. Once I completed everything, I submitted my pull request and carried on

      I actually had fun working on these projects, especially after the mess that was hacktoberfest. Hacktoberfest was very stressful with all the finding issue fiasco and getting everything done weekly. This time, it felt more relaxing since I had more time to work on everything, including the external project issue. As for the next assignment, I honestly don’t know what else I can possibly work on. Hopefully, I find something very soon because participating in open source projects is actually fun and worth doing in my spare time

      by Timothy Morris at November 16, 2019 05:34 PM


      Sefa Baah - Acheamphour

      Release 0.3 Completed !

      From the beginning to now; of taking DPS909, release 0.3 has been the most frustrating period. This doesn't have anything to do with writing code or with issues, no. The most frustrating has to be waiting for my issue to be reviewed and watching my code have a conflict with what is in the master on gitHub.

      Backtracking for a bit, the main idea of release 0.3 is to have two pull requests, one has to be internal and one has to external. The internal was to have any improvement in our course's blog feed website; planet cdot. The external had to be an issue we found on GitHub like before expect it had to have a more meaningful contribution. 

      Internal

      The issue I chose to solve was to find a way to have the website be able to translate the text into any language. To find out how to find a way to solve this issue I turn to my favorite helper google. I found some sample code that does what I need perfect but it was powered by google. What I later found out google stopped supplying this service but allowed old sites that had it to still keep the service. So I had to scour the google search to find an equivalent and I found G|translate. It a website that allows it's customers to translate their website with a subscription from G|translate but I able to use their free version.


      The code above is what allowed me to give the website the functionality to translate default text to any language that appears in the drop-down box. The problem I had was after I had created a pull request after time has passed I will have a conflict between my code and that was in the master. At one point the whole folder layout has changed and I had forcibly get the telescope project from the master because my version was out of date.

      External

      For the external, I decided to finish an issue I agreed to complete during the hacktoberfest month but I had trouble with it. The project was written in PHP and since I had to experiences with PHP I was in a bind and with the month I decided to push it to my next release.  So here we are with this issue, it was to change the color scheme of a donut chart depending on the value the data represented.

      I had to research how to write PHP and the most trouble I had was to set up the environment. Since I was having trouble I sent a message to the project's creator to ask for help setting my environment. And boy it was an opener since PHP required a server to actually run, I had to download dependencies to get started plus one more dependency that I haven't heard of. Composer, it is an application-level package manager for PHP, provides a standard format for managing dependencies of PHP software and required libraries.

      This fix required me to edit two views dashboard.twig and network.twig,  the trouble I had was to figure were had to do the actual editing when I started this project. I had to do a search looking for specific keywords to find those two files.



      network.twig
      dashboard.twig

       This has been a nice experience, not great but nice. The reason was that when I made a pull request was merged or taken for a review but working on this internal part has been frustrating because every time I go the page to check on my pull request there will conflict that I need to resolve.

      External
      Internal


      by DPS909 - Sefa's Blog (noreply@blogger.com) at November 16, 2019 05:20 AM


      Ehsan Ekbatani

      Release 0.3 - Pull Requests Telescope & LifeStyle Ecommerce

      In my previous post, I revealed the issues I will be attempting to resolve in order to contribute for an internal project and an external project. I also detailed the projects themselves, so if you haven't already read my previous post, I would advise starting there to give my current post some context.

      I can successfully say I reached the goals I set for both the external and internal project in my previous post. Keep reading to find out how I did this, and the challenges I overcame.

      Internal Pull Request

      Adding Pagination to CDOT Telescope

      The responsive page navigation is circled at the bottom

      This issue was assigned to me by a colleague and was much trickier than I initially anticipated. The issue simply required to implement paginating functionality to the front end, so that the blog posts are displayed in pages instead of all  being displayed in one very long scrollable feed.

      The first challenge came with the starting point. The home page was designed with a very specific bootstrap library (MD Bootsrap) that had some different conventions from traditional bootstrap, therefore I had to pull up the documentation for that library to figure out how to design the page navigation. 

      From there I proceeded to implement the functionality in a JS file called Pagination.JS using jQuery. Since the original contributor used a CDN link to jQuery it was not recognizing any of my code initially. After further investigation, I found that the script tag calling the library MUST come before the script tag calling my file in order for it to recognize my jQuery. 

      The next challenge was making sure it is implemented in a modular way so that it can be easily configured for how many blog posts per page by simply changing the value of the "limtitPerPage" variable. 

      The final bug I bumped into was the eslint linter test. It was initially not recognizing jQuery so I had to add that in it's configuration file, but then it was still throwing errors. These errors involved things such as using "++" operators, or having strings and comments over 100 characters per line. Since I was dynamically creating HTML tags, some strings needed to be long. 

      My first approach was to change the rules to correct this problem, but the community rejected this solution as it would conflict with the selected AirBnB standards. So at this point I learned that it would make more sense to change my code, rather than the rules to pass eslint's test.

      External Pull Request

      Re-Building LifeStyle Ecommerce as an Angular App

      This was my biggest issue I've taken to date and it was as challenging as expected. Although it is a smaller project (5 Contributors), I gravitated towards this because it is a front-end theme meant to be used a starting point for web based e-comemrce projects. As a former freelance web developer I always appreciated this kind of projects as expedite the process and provide great starting points.

      As I expected, the major challenge here was going to be in re-implementing the logic in typescript. The original app is a basic HTML, CSS, & JS stack with the use of common libraries like bootstrap. These all translate well into angular so re-building this isn't too difficult, just a little tedious. While Javascript (ES6) is syntactically very similar to Typescript, it cant be simply copied over in a singular file. This is mainly for the logic that dictates the state of the app, for example, items being added to the cart while shopping, would have to be implemented in the '.ts' file for the cart component due Angular's component oriented architecture.

      Since Angular apps are single-page applications, each page had to be reconstructed as it's own component and dynamic elements of that page were constructed as their own components. Ultimately I managed to achieve output and behavior as the original website.

      UPDATE (Nov 15): I have received some feedback and requests for changes from the project author. The first request seems to be in regards to alignment issues in some HTML files. This likely occurred when I had to change some of the custom tag names from the original since Angular's parser does not recognize custom tags by default. The other was in regards to an unused component I created called "lists". I made this initially when trying to "componentize" the website, however never ended up needing it and thus forgot to remove it.

      I have made these changes and requested a re-review.

      UPDATE (Nov 16): A final change has been requested by the project author. Since I have three identical consecutive commit messages, he requested that I be more specific in them or squash them if they are not significant. In retrospect, I agree that I should have been more specific in the messages, so I will be sure to do that in the future. I will squash and hopefully have my contributions merged soon.

      Next Steps for Improvement

      As mentioned earlier this was my biggest, most challenging issue I've ever tackled as an open-source contributor, however the project was quite small so that's where I should look to improve. 

      My next major milestone would be to take on an issue on a big and popular project. Maybe not an issue of this magnitude that entails re-implementing an entire website, but certainly something in the ball park of adding or enhancing some functionality.

      Stay Tuned for the final release!   






      by eekbatani (noreply@blogger.com) at November 16, 2019 04:59 AM


      Yohan Choi

      Release 0.3 - Part 2

      External Pull Request


      In the previous post, I mentioned that I will solve the blurry font rendering issue in Hyper. To reproduce the bug, I resized and tested the window of Hyper.

      I found that the issue is related to the followings:

      1. The issue is not related to the width of the window, but it is associated with the height of the window.

      2. When zooming in/out screen, the issue happens in different conditions base on the font size.

      3. The height is different based on the monitor size.

      4. The issue does not happen when the screen mode is full-screen.

      5. The issue happens only in Linux, not windows.

      <Before>

      <After>
      The height of the first one is 1017 pixels, but the height of the second one is 1044 pixels.
      66 x 56 66 x 57

      As you can see, the first screenshot is very clear, but the middle of the second screen is blurry. Perhaps, this issue may be related to the maximum size of the windows. So I decided to investigate the config files.

      Since Hyper is run on an electron, I guessed this issue may be related to electron, so I searched the issue with an electron, and I found out that this bug happened in another project also. I gain some clue on electron issue-13646

      However, Hyper works fine in another environment, and if the issue is related to the electron, I should join the electron, which is out of scope. So I decided to find a way to solve the issue within Hyper. The first thing I did was to play the config file again.


      config: {
      // choose either `'stable'` for receiving highly polished,
      // or `'canary'` for less polished but more frequent updates
      updateChannel: 'stable',

      // default font size in pixels for all tabs
      fontSize: 11,

      // font family with optional fallbacks
      fontFamily: 'Menlo, "DejaVu Sans Mono", Consolas, "Lucida Console", monospace',

      // default font weight: 'normal' or 'bold'
      fontWeight: 'normal',

      // font weight for bold characters: 'normal' or 'bold'
      fontWeightBold: 'bold',

      // line-height as a relative unit
      lineHeight: 1,

      // letter spacing as a relative unit
      letterSpacing: 0,

      One interesting was that font-size is related to line-height. In other words, if I change line-height, the font does not look blurry anymore. But proper line-height for each font size is different. I guessed that the issue may be related to the CSS issue, and I found out that sometimes text looks blurry in the Chrome browser when using transform property. But it was too complicated to fix transform property since I had to fix the whole program.

      I played with the configuration, and I found out that when font-size is an odd number, the line height should 1 to render text clearly, and the font-size is an even number, the line height should be greater than 1.1 to render text clearly. When zooming the window of Hyper, the Hyper actually change the font size. To fix the issue, I had to make update the font size with line-height together, and I found  ui.js and add three lines. If the font size is even number, it adds line-height. Then also through zoom in and zoom out the window, it updates the line-height together.


      if (config.fontSize) {
      ret.lineHeight = config.lineHeight;
      if(config.fontSize % 2 === 0){
      ret.lineHeight = config.lineHeight + 0.1;
      }
      }








      As you can see, there is no more blurry text even zooming in/out or change the size of windows, and I made a pull-request. Perhaps, this pull request will be reviewed, but it was a good experience to solve an unknown problem.

      Internal Pull Request 2


      The second pull request is related to front-end design. It seems like we don't have normalize.css and grid system. Adding normalize.css makes browsers render the website more consistently in different environments. The responsive grid system helps the webpage becomes responsive. Adding these necessary CSS libraries will help the web site rendering.

      Also, I added the border-box property, which makes no adding padding and margin to box. It helps to design the webpage easily.

      * {
          margin0px;
          padding0px;
          box-sizingborder-box;
      }

      Well, I'm also into the front-end part, but it seems like there were a lot of changes. But it does not look like being organized well. Now, we are focusing on mockups for the front page, but I think that it will have trouble in the future. In release 0.4, I will focus on the design part. To be honest, we have not discussed the design. For instance, the navigation panel is on the left side, but it makes it uncomfortable in the mobile environment. As a result, the web site needed more code to render on mobile. If the navigation panel is on the top, it will be easier to implement. Plus, there is no search bar so it makes uncomfortable to search for information. In the next release, I will focus on the UI and UX part.


      Well. joining open-source projects are a really exciting experience. However, it would be good to have more time to dig into the project. But learning about git and corporating with others was a meaningful experience.



      by Yohan (noreply@blogger.com) at November 16, 2019 04:47 AM


      Carlos A. Gomez

      Of front end changes smalls and big

      Last time I only did small changes the HTML project, today with a new front end I found issues that relate to both accessibility and stability.

      When experimenting with the new front end, I noticed our close button was not responding as expected on mobile, this issue is surprisingly normal, as areas of effect work differently from phone to phone, you should expect this. That said the button was a bit too small for both noticing- if you believe it my parents did not notice close buttons, and one struggled to read,

      The obvious answer was to make it larger. So we went from this?

      To this


      You will notice, the buttons now have a larger readability on a phone, as well, as having a better structure to be able to be pressed and read. A side by side comparison helps to visualize this a bit better

      you can see it in action here

      https://cagomezr.github.io/telescope-demo/

      I did howver start prototyping a new version with specific changes on html and css

      I am now prototyping a new version? You may ask, well I notice a few issues aesthetically, and in some ways code wise. is a bit strange to explain code wise so Ill do my best

      Absolute positioning is an item most developers where i used to work would frown on, unless absolutely necessary. Why? Because we have the ability to position relatively, and absolute is basically overriding and you are opening yourself at having to change an item if something on the UI changes, absolute is mostly focused for “ modal popup” which could well be a function we look into the future of this for what ever reason, maybe views login I would like to keep the option of keeping our absolute functionality to the bare minimum.

      Also one of the reasons I decided to also change the structure was because, the way the structure of the menu? It takes a little bit more of the user by forcing mid screen, this way we can make the user less effort and a bit more accessibility. (it sounds small but you be surprised by how this little changes fix engagement so we needed with a model that changes the button structure to stop using absolute and we change the flex to be able to start from top.
      The result can be found here on the prototype. Once again you can see the changes from the one currently approved,

      A side by side comparison for reference

      the buttons are moved up, the close button is slightly larger, AND is not absolute,
      the changes are basic we only move one line, and we switch the CSS justify-content to flex-start in the nav CSS. You can see this proposed changes here in

      https://cagomezr.github.io/telescope-demo/indexchangeproposal

      Regardless of what this pet project changes bring I m thinking of a few additional changes to structure, of HTML to add favicon and make sure that this goes well over all front-end 0wise.

      My external did not go well I admittedly choked trying to find an issue, It was hard, until I found a bug in a project that i helped before that I am interested with Geo Locations This unfortunately came to a close due to the fact the database to project seems to have issues, and I don’t think I can finish, that said, I accept that, my plan for this protect was to figure out maps with react, something that I have never fully worked with and I was curious about as I have done some geolocation in a very basic concept.

      https://github.com/cagomezr/unescomaptesting (for reference about what I was working on)

      and I was looking forward to working further with this, the best i could do was lo an issue with project due to Firebase issues

      https://github.com/chrisdesilva/pickup/issues/56

      the issue I planed to work with was https://github.com/chrisdesilva/pickup/issues/34 which dealt with changing the pins and also adding the functionality to change this, I’m disappointed in myself if I m honest.

      but I think I only have to blame myself for this, teaching you learn from my mistakes, and try to ask for help. and dont be too proud to ask of it. Yes I know do as I say, not as I do but what can I say..,

      Im only human.

      I think.

      by cagomezr at November 16, 2019 04:33 AM


      Sarika Hanif

      Release 0.3 — External Issue

      Release 0.3 — External Issue

      For my external issue for release 0.3, I worked on the AncientBeast repository. This project is a turn based strategy game that is written mainly in JavaScript. When I first started to work on this project I ran into a couple of issues with the environment. Following the documentation I was able to run the project, but any changes that I made to the code didn’t affect the game. I deleted so many functions and they still worked in the game! I eventually noticed that the documentation mentioned gitpod. It pretty much builds the environment and you can edit the code. It was pretty cool to test out and I used it to test my changes.

      Gameplay of Ancient Beast

      In the issue that I worked on, it was suggested to modify the hotkey functionality for the scoreboard and dash view and also to add a hotkey for fullscreen mode. The requirements were to use Shift + S to open the scoreboard, Shift + D to open the dash board and Shift + F to toggle fullscreen mode. Originally, pressing the g and t key would toggle the scoreboard and dashboard. After looking through the code, I found that the functionality was found in the interface.js file. There was an object created called hotkeys. This stored the ascii values of the hotkeys associated to functionalities in the game.

      let hotkeys = {
      scoreboard: 84, // T : This toggles the scoreboard
      overview: 71, // G : Toggle dash
      cycle: 81, // Q : Switches between usable abilities
      attack: 87, // W
      ability: 69, // E
      ultimate: 82, // R
      ...
      }

      Then iterating through the list, the KeyDownEvent is listening to see if shift + whatever value in hotkeys was pressed. It will then take the key associated with the value and preform the action within the switch case.

      $j.each(hotkeys, (k, v) => {
      if (e.shiftKey && v == keypressed) {
      switch (k) {
      ...
      }
      }

      In the switch case, all we need to call is the triggerclick() on the buttons for scoreboard and dash view.

      // Score Button
      this.btnToggleScore = new Button({
      $button: $j('.togglescore'),
      click: () => this.toggleScoreboard(),
      },
      game,
      );

      For the fullscreen functionality it wasn’t as easy. There was a fullscreen button in the script.js file, but I wasn’t sure the best method of implementing this. In the script file I exported the following functions:

      export const isNativeFullscreenAPIUse = () =>
      document.fullscreenElement || document.webkitFullscreenElement || document.mozFullScreenElement;
      export const disableFullscreenLayout = () => {
      $j('#fullScreen').removeClass('fullscreenMode');
      $j('.fullscreen__title').text('Fullscreen');
      };
      export const enableFullscreenLayout = () => {
      $j('#fullScreen').addClass('fullscreenMode');
      $j('.fullscreen__title').text('Contract');
      };

      isNativeFullscreenAPIUse uses document.fullscreenElement which is the standard syntax that returns the displayed element in fullscreen mode or null if it’s not fullscreen. The other two properties are for chrome, safari, opera and mozilla. The other two functions modifies the styling of the code within the fullscreen id. In the switch case, I checked to see if isNativeFullscreenAPIUse was true if so, then exit fullscreen, else request fullscreen for the game.

      Now the weirdest part of this was that the fullscreen wasn’t working in my browser. It seemed as though fullscreen only zoomed into one portion of the game. It looked like this on my laptop:

      I was unsure of how to fix this, I spent a lot of time trying to fix it. I tried messing around with the css when the game entered fullscreen by using :-webkit-full-screen and :fullscreen but it didn’t work. Wasn’t sure if I need to somehow shrink the background, players, grid etc. I asked the author and surprisingly, my changes worked fine when he tested it. We are both running the same versions of chrome too. Strange, looks like we found a new issue.

      After reviewing the changes, my PR was accepted.

      This was a fun project and I would like to contribute to this again!

      Onto release 0.4 ✌🏼

      by Sarika Hanif at November 16, 2019 03:43 AM


      Krystyna Lopez

      Release 0.3 - Work that's done

      In the previous blog I was describing issues that I was planning to work on this week. Indeed I saw many different issues that I would like to contribute on. In the internal project Telescope I was working on Issue that suppose to update Stylelint in order to check only .css files.




      After few hours of reading through and working on this issue I send my PR. I was also able to create a new issue so I can keep working on this project and contributing to it. As I said before, to work on the Telescope is new experience for me and I hope with professors supervising me and my classmates will be able to create a nice project.


      For my external issue I was thinking to work on Wordpress/Gutenberg  project but in the end I change my mind and decide contribute to Microsoft/STL. There are many issues that has to be fixed on this project and looks like not too many people have time to help this project solve issue. This week I picked this issue. I'm very glad that I was able to work on it because issue has been created by member of this project but when I submit my PR some other members did not consider that should be an issue at all because resolving issue that I work on will break the complication of the code. Finally members of the project run the test and decide that changes will simplify the code without confusing users of the code.




      by Krystyna Lopez (noreply@blogger.com) at November 16, 2019 03:17 AM


      Matthew Clifford

      Internal/External PR’s

      This week I worked on 2 different Open Source projects. The internal open source project at school for the Planet CDOT website , and also a external open source project in github.com/microsoft/vscode.

      I had noticed an open issue that wanted all of the ‘Edit in settings.json’ clickable texts in the settings editor to become buttons as it would be more obvious while tabbing for accessibility users..
      I noticed above that there was a ‘Add Patern’ button int the settings editor already, and so I decided to replicate that button, but, for the ‘Edit in settings.json’ text.

      old version

      new version (my pull request)

      The internal open source project I worked on some of the documentation , and also a function which extracts URLs from a blog post’s html text and then returns an array of the URLs that were extracted.
      Originally I had written this function in python, but I was asked to re-do it in javascript using nodejs..

      PR’s :
      1) https://github.com/microsoft/vscode/pull/84946
      2) https://github.com/Seneca-CDOT/telescope/pull/28
      3) https://github.com/Seneca-CDOT/telescope/pull/102

      by smilegodly at November 16, 2019 12:10 AM


      Jatin Arora

      Release 0.3: The Finale

      This week was the final week to get done with my Release 0.3 work and I did.  Some claps for me. But this was not like my usual assignment in a course. It was something that I had to decide, commit to and get it done. The work I wanted to do was chosen by... Continue Reading →

      by jarora4 at November 16, 2019 12:01 AM

      November 15, 2019


      Julie Philip James

      Release 0.3: Completion

      In my previous post, I had discussed briefly on the issues I was planning to work on for Release 0.3. I worked my internal issue (telescope) where I had to configure jest – a javascript testing framework so that it provides a coverage report for the tests. The coverage report would present an analysis of the number of statements, functions, branches, and lines of the code tested. I configured jest within the package.json by adding this piece of code:

       "jest": {
          "collectCoverage": true,
          "testEnvironment": "node",
          "coveragePathIgnorePatterns": [
            "/node_modules/"
          ]
        } 

      collectCoverageindicates whether the coverage information should be collected. Our testEnvironmentis node andcoveragePathIgnorePatterns indicates that coverage information should be skipped for the file. This what ‘npm test’ would give us with this addition:

      Thankfully, my code was merged.

      issue | pull request

      My external issue was to change virtual functions into overriders. This was an issue I found in Microsoft’s STL repository. I had to lookup virtual and override functions again. I picked to work on a single function – _Doraise() because the code was too big for me to alter every function. I had to identify base declarations of the functions and add override to the functions within the derived. The pull request I initially submitted did not pass all the tests because of spacing errors. Also, my commit message for the pull request was wrong. I struggled with git to refresh what I had earlier learned about squash, rebase and amend. But thankfully that worked too and I got positive feedbacks from the code maintainers.

      issue | pull request

      by JULIE JAMES at November 15, 2019 11:11 PM


      Igor Naperkovskiy

      Contribution to telescope

      This week I was working on a wiki parser for wiki page. Its a new project that was started by Seneca to update their existing Planet. Its suppose to track blog feeds, that are written by Seneca students and staff. My part was to create a parser that was suppose to parse a wiki page and extract blog urls and the names of students that wrote these blogs. Since its the first time that I worked with node it took me some time to get used to. One of the things that was new to me is a Promise. Had to do some digging on how to extract information from a Promise. Also used some new packages like bent and jsdom. Overall it was a great learning experience and I learned a lot from it.

      I have also done a contribution to Gatsby project, by translating one of their pages to the Russian language. This was the first time I ever done a translation, since the languages are quite different and I couldn’t just translate word for word, I had to remember a lot of Russian statements that I haven’t used in a while, but in the end it worked out quite well I believe.

      by soltylis at November 15, 2019 10:50 PM


      Sina Lahsaee

      November and Open source

      During November I will be tackling two things at the same time. I will looking for an external project in the open source community that will be challenging me more than what I did during Oktoberfest, as well as taking on an internal project with Seneca College for their CDOT planet named Telescope.

      Telescope

      Telescope is an open source project maintained by David Humphrey and his fellow Open Source students at Seneca College in Canada. This project focuses on updating the Planet CDOT website by upgrading visuals, containerization, as well as incorporating some aspects of the machine learning such as "Read time per article". The main frameworks being used so far are JavaScript, CSS, HTML, Redis, and Docker

      For the telescope project, I decided to tackle on the front-end aspect of it by changing the design of the website. Working along-side wonderful colleagues, mock-ups were made and based on those I worked on developing a static HTML page closely resembling the mockup. Here's a link to the git-hub page website for preview purposes only, and anyone will be more than welcome to make suggestions. Keep in mind that this project is in its early stages and many improvements can still happen. 

      External Project

      Unforunately, due to a nasty flu, I have not been able to find a desirable external project that has me interested so far, But I will be sure to update you guys on the progress made in the upcoming week. 

      Hope to see you guys next week. 



      by slahsaee (noreply@blogger.com) at November 15, 2019 09:35 PM


      Ryan Wilson

      Open Source Contribution Update

      As mentioned in my previous blog post, the assignment we were assigned was to file a PR for our internal and an external project. For our internal project our PR must be merged to count and for the external PR we must pick an issue that is larger than anything we did in Hacktoberfest. In my opinion this assignment was definitely more challenging than the previous assignment of submitting one blog post and one PR for Hacktoberfest.

      Internal PR

      For the Telescope project I created a basic email sender using Nodemailer which will send a message to a list of recipients with the desired message content. I really enjoyed to research and implement the different email senders to test to see which one is the most suitable including Nodemailer and SendGrid. By the end of that conversation, I decided to use Nodemailer because SendGrid could not send to the Seneca email servers as it refuses the connection.

      This pull request consisted of many messages back and forth to change and add to my initial PR.

      31 Messages

      At first when I created this sender, it was used to send from a gmail account. This is a portion of initial code

      const transporter = nodemailer.createTransport({
      service: ‘gmail’,
      auth: {
      user: ‘’, // Email Name (.env variables must be used refer to issue#60)
      pass: ‘’, // Email Pass (.env variables must be used refer to issue#60)
      },
      });

      After discussions, environmental variables were added to the .env/sender file, promises were added, return type was changed from info to info.accepted for other issues and the service was switch to an SMTP server.

      This is what the final code looks like

      Step 1
      exports.sendMessage = async function (receipiants, subjectMessage, message) {
      return new Promise((resolve, reject) => {
      const transporter = nodemailer.createTransport(‘SMTP’, {
      host: process.env.NODEMAILER_SERVER,
      port: 25,
      secure: false,
      auth: {
      user: process.env.NODEMAILER_USERNAME,
      pass: process.env.NODEMAILER_PASSWORD,
      },});
      Step 2
      transporter.verify((error, success) => {
      if (error) {
      console.log(error);
      } else if (success) {
      console.log(‘Server is running properly’);
      }});
      Step 3
      const mailOptions = {
      from: process.env.NODEMAILER_USERNAME,
      to: receipiants,
      subject: subjectMessage,
      html: message,
      };
      Step 4
      transporter.sendMail(mailOptions, (err, info) => {
      if (err) {
      reject(err);
      } else {resolve(info.accepted); }});});};

      How it works

      The above code is fairly simple,

      • Step 1 — Create the transporter which is used to connect the services
      • Step 2 — Used to verify the connection for the SMTP server
      • Step 3 — Message Content
      • Step 4 — Send the message with the message content and return a promise if it fails or succeeds

      The tests for nodemailer are created in another issue — https://github.com/Seneca-CDOT/telescope/pull/198

      Process

      Nodemailer comes with really good documents as you can find a template for the code. The only process that is involved is which service you want to use whether its an email or SMTP. After you figure that out, you can configure some of the components such as which info to return when successful (example: info.accepted will return an array of recipients that received the email)

      Experience

      My experience creating this first pull request was pretty good as I learned as I made many different mistakes. The most important thing that I got out of this, is that sometimes it takes many days for others to respond which can make it slow to complete a task even if its small. When your relying on others so you can get the feature merged, it was frustrating when you have all of the approvals but your package.json has conflicts. This means that you must git pull upstream master, then rebase into your branch and get their approvals again which may take hours if others are inactive. I added an explicit variable into the Nodemailer transport but it does Travis Ci reject. In PR #227, I removed this explicit variable.

      External PR

      My external PR was from one of the projects that I completed a PR for on Hacktoberfest. The project that I choice was Darkreader and the issue was that the shortcuts text duplicate when you click on the them and click on the toggle buttons.

      Issue

      Below is an image of the issue that occurs.

      Exact steps to replicate this issue.

      Issue is on Firefox

      1. Click on the either “Toggle current site” or “Toggle extension”, this text will change to “…⌨”
      2. Click any of the buttons (toggle buttons, navigation buttons)
      3. The shortcut button that you clicked on should have duplicate text

      Preparation

      The code base for Darkreader was huge and I did not know where to start.

      I started from the structure of the code. When you first grab the app from Github, you must run two commands NPM Install and NPM Run Debug.

      The NPM Run Debug automatically generates two debug folders, one is for Firefox and the other is for Chrome. So i started to look around the debug folder created for Firefox. After searching, it seems like it will be in this popup index.js file as its an issue with the component on the UI page. My first impression is that this one file is huge as it contains 3000 lines of code, but its probably not that big compared to some of the other software that we use on a daily basis.

      The default for these shortcuts are “Alt+Shift+A” and “Alt+Shift+D” so i tried to find the instances in this file.

      Find instances of shortcuts

      I put a breakpoint and started to walkthrough the program to try to understand the code.

      First Function
      Second Function
      Third Function

      After this function, It runs through many more functions before it actually sets the node in the below function

      After hours of walking through the code, I still had no idea what was changing text. So i decided to play around with the code and stumbled across this function.

      Now what does this do? Basically after clicking any elements on the UI it reinserts the nodes. I assume it checks to see if anything is mismatched and if it is then it insertNode.

      So it reinserts the nodes, but does not actual set the text in this function. So maybe if it finds multiple instances of the Toggle text then do not reinsert and keep it the same.

      I added this if statement which prevents the shortcut from being reinserted and it works. After I finished, I realized that you can not submit the files in any of the debug folders which means that I made zero progress in solving this issue after many hours.

      I tried to find this insertNode function but it did not exist in the actual files. I found out that all their Node functions are generated by a Javascript framework called Malevic.js from the below image.

      Back to the drawing board, I look through the source files to see if I can find anything for the shortcuts. Founds the shortcut folder and the index.tsx that comes with it.

      I set a breakpoint to see what each function does

      This function set the shortcut text and if you press a button down while your focused on it, you can change the shortcut

      This is the off focus function, after you click off. The purpose is to set the text back to the original text.

      Combining my knowledge from walking through the debug file and walking through current files. When you click on the shortcut and click a button, it sets the text to the original text in the onBlur() function and then in the matches function it adds onto the text message because node.textContent = “…⌨”

      I played around for a few days in the onBlur function to see if i could catch to see if Toggle appears more than once, but the data coming into that function is inconsistent.

      Data that comes back

      Toggle extension Alt+Shift+D
      Toggle extension Alt+Shift+D
      Toggle extension Alt+Shift+DToggle extension Alt+Shift+D

      After playing around with the onBlur() function it was impossible to find a solution because of the initial state and second state being “Toggle extension”.

      After a while I found that every time it calls the InsertNode() function in the debug folder it calls this function

      Solution

      const shortcutClass = document.getElementsByClassName(“shortcut”);
      const toggleCurrent = document.getElementsByClassName(“shortcut”).item(0);
      const toggleExtension = document.getElementsByClassName(“shortcut”).item(1);
      if (shortcutClass.length == 3) {
      if (toggleCurrent.innerHTML.lastIndexOf(“Toggle”) > 5) {
      toggleCurrent.innerHTML = toggleCurrent.innerHTML.slice(0, toggleCurrent.innerHTML.indexOf(“Toggle”, 1));
      }
      if (toggleExtension.innerHTML.lastIndexOf(“Toggle”) > 5) {
      toggleExtension.innerHTML = toggleExtension.innerHTML.slice(0, toggleExtension.innerHTML.indexOf(“Toggle”, 1));
      }}

      Just a brief summary as this blog post is pretty long. I created three constant variables which represent the two shortcut buttons

      It comes into this function on the initial creation which means that it inserts the nodes one by one onto the UI and also when it refreshes the UI in the matches function.

      I check to see if the data coming back contains at least 2 instances of Toggle which means the data represents (Toggle extension Alt+Shift+DToggle extension Alt+Shift+D).

      If 2 instances of text occur, it then slices the text to take the first instance and then sets the shortcut text.

      Experience

      This PR was a wild ride, it made me realize that a small bug can mean that you have to search through and understand a lot of the code. The most important lesson that I learned is that you should change the file and save it to see if you can make commits to it before spending too much time finding the solution. I really enjoyed to see the coding styles and also trying to understand how to code works. Its really hard to understand the code when a portion of what your working on it generated by a Javascript framework but after you understand enough of the code, it becomes easier. Walking through the code with the debugger is probably the best things that I did to prepare myself for this PR. I now understand most of the code that goes into the shortcuts on the UI and a general understanding of the entire UI which will come in use in future PR’s.

      Links

      External
      Issue — https://github.com/darkreader/darkreader/issues/1621
      Pull Request — https://github.com/darkreader/darkreader/pull/1732
      Internal
      Issue — https://github.com/Seneca-CDOT/telescope/issues/25
      Pull Request — https://github.com/Seneca-CDOT/telescope/pull/93
      Previous Blog Post
      https://medium.com/@rwilson31/open-source-as-a-class-68c7434c83a1

      by Ryan Wilson at November 15, 2019 06:41 PM


      Paul Luu

      Continue working on dungeon

      Continuing from the last blog, I will be contribute dungeon again. I find the project to be very high level (using tools like Travis CI, Maven), but there is only one maintainer for the code. I wanted to work on bigger well known projects but with other courses (SPO600 taking the most amount of time) to handle I have little time to branch out.


      Sadly, I had some problems with building the project. Previously, I was working on this project on a different computer but I accidentally wiped that computers hard drive (Luckily, I didn’t lose anything that important). Installing the correct programs needed for the project and running the commands I thought the project would build fine but I was present with this error during the command mvn test:

      Failed tests:   testResourcesFolderContainsTheSplashScreenFile(org.mafagafogigante.dungeon.io.ResourcesFolderTest): Resources file should be a directory
        testResourcesFolderContainsTheCorrectNumberOfFontFiles(org.mafagafogigante.dungeon.io.ResourcesFolderTest): Resources file should be a directory
        testResourcesFolderContainsTheCorrectNumberOfJsonFiles(org.mafagafogigante.dungeon.io.ResourcesFolderTest): Resources file should be a directory
      

      I never seen this error before, and I am pretty sure I did not edit the resource folders. There’s been another merge from a different contributor since I last work on this project but looking through the pull request nothing there suggests a change of resource folders. I decide to run this test on Travis CI. I have never used Travis CI before so this was a really good learning experience. Running the test on Travis CI passed so I decided to ignore the issue and continue with implementation. Updated: I later found out that this issue is because one of my directories had a space in it, which causes this error. Reference this issue for more details.

      Issue

      Looking through the issues, most of them seem to require a lot of effort and will probably take more than the allotted time to complete. Luckily, I found an issue that I feel like I would be able to implement. I had some ideas on how to implement it but the issue still seemed to be challenging.


      The issue I chose was to implement a map legend for the game. Currently, while the player is traveling there’s not much indication of where you are or what’s around you unless you use the map command. It would be really usefully if there was a map legend on the bottom right corner of the screen to give some idea where you are on the map.

      Ideas for implementation

      I initially had two idea on how to implement the solution once I saw this issue.


      The first solution would be to add another text area box at the bottom right screen of the window. This means using Java Swing which I have done before with my Second pull request for Hacktoberfest. In my blog I mention how difficult it is to code with Java Swing since the layouts are troublesome to learn. Overall I think the second solution is better.


      The second solution would be to add the map to the already shown text area itself and make sure it shows on the bottom right corner of the screen. This may be more work, I have to count the amount of lines that already in the text area, and work out where to place my map text on the bottom right corner of the screen. Below are the images of the game and a mock-up of what I want the final result to be:

      What the game looks currently:

      What the map command looks like:

      The @ sign represents the player. The characters represent a location. (For example, F stands for forest). The ~ sign represents unexplored areas.

      Mock-up of map legend using paint:

      Commenting on the issue to ask if this is what the maintainer want, to my surprise it wasn’t. Turns out I mistook map legend with mini-map, what he actually wanted is for the symbols to be defined like so:

      Implementation

      I also had many ideas with how I would implement this but as I was implementing it I ran into many problems. I decide to reuse some existing functions but the result is not exactly what I wanted:

      On first examination it seems to be fine but I listed out the problems here on my pull request, where I ask for aid.


      Running mvn test and fixing code convention errors I decide to upload this commit to get feedback and potential help from the maintainer.

      Asking for help in my Pull Request

      I commit my changes and create a pull request. The purpose of the pull request was to address some of the problems I had and see if I can get some help. I will continue to keep working on the issue on my own, but I wanted some feedback to know if I am going in the right direction. Update: After getting really good feedback from the maintainer I decide to take his suggestion and ideas and keep working on the problem. For more information see the pull request

      Another pull request

      Creating a new pull request with his suggested ideas and changes from the previous pull request . I would explain my thought process but the pull request already does that. The finally result looks something like this:


      I really do want to branch out to more well known projects but I truly am busy with other courses. Hopefully if I get a breather I can try to branch out to a new project but if not I will probably be contributing to this project again.

      Links:

      by pluu3 at November 15, 2019 06:02 PM


      David Medeiros

      Release 0.3 - Third time's the charm?

      OpenClimateFix - A New Issue

      Here we are again. I have bounced from issue to issue, hitting roadblocks as I go. I know that the issue I have picked here is not as complex as tania-core, but I believe it is a little more complex than WritingSwarm and is in a system I am familiar with. I am devoting all of my available time to completing this issue, and I intend to have a pull request up by tomorrow night. In order to do that, I will have to work hard in order to get this done, and I am committed to doing that.

      The Issue

      The issue I have chosen is to create a contributors showcase component, for a new Get Involved page on the openclimatefix website. I have worked on openclimatefix before, as my last pull request for Release 0.2. In order to populate the contributors showcase, I must read a json file containing each contributor and display their name, profile url and github avatar.

      Setting up the Environment

      I already had the environment set up as I have worked on this project before. It uses yarn to install dependencies, Gatsby as a framework for development, and a host of dependencies to run tests and ensure consistent formatting. I updated my fork of the repo and created a new branch to work in. 

      In terms of research, although I have used Gatsby before, I have never created a new component. I had to figure out the conventions used in the openclimatefix site and copy those conventions. I also had to figure out how to read a static json file in Gatsby. Turns out you need an extra plugin, called gatsby-transformer-json in order to read in json files using graphql. graphql comes with Gatsby and is used to manage static and dynamic data in a website. I also had to learn what a StaticQuery is in Gatsby, in order to display the results of the graphql query. 

      The Solution and Pull Request

      The Component Code

      Component Display Code
      This function returns the Contributors header with each contributor displayed below it. The StaticQuery has 2 properties; query and render. query uses graphql to query the json file named contributors.json and get the contributors attribute's contents, in order to create a javascript object containing each contributor's name, profile (url), and avatar (image url). The render property will run a function to render the data generated using the function getContributors().

      getContributors()

      getContributors() receives the object generated by the graphql query
      getContributors() receives the data from the json file and for each contributor makes a <div> element to contain the contributors profile image, and their name, which is also a link to their profile. In order for the render to work, a key must be chosen, so I used the name property for this key. For each contributor in the contributors array, a div with their information will be appended to the cont array, and return to the component code. This will populate that code with each grid item, ready to be displayed.

      CSS

      Basic CSS to set up the grid

      Css works a little differently in Gatsby than it does in regular HTML5 development. It is called like an object, with each class represented and called to as an attribute of that object. Whenever you see "styles.something" in the className properties above, they link to a class in the CSS document.

      Final Result


      Interactions with Flo (FWirtz)

      Flo is the maintainer of the repository, and is very helpful and communicative. She and I have not talked much during this process, but she did assign me to the issue. We have talked before on my previous 2 issues in openclimatefix,  and I expect to hear from her about this pull request, although I don't think she has reviewed it yet.

      Release 0.3 - Overall Experience

      I felt very rushed in this release. We only had 2 weeks really, considering Release 0.2 was due Oct. 31st and I needed to work on it until then. I had other commitments, and as hectic as October was, November has been worse so far. There are only about 4 weeks left in the semester, and in that time I have to complete Release 0.4. Although Release 0.3 was difficult for me, and I'm not sure how well it went, I know that I'm much wiser and more aware of my abilities. For one thing, I am going to avoid databases for Release 0.3. Secondly, I am going to find a good issue early on, and get started on it right away. 

      by drwm-base (noreply@blogger.com) at November 15, 2019 05:40 PM


      Hansal Bachkaniwala

      Contributing to Telescope

      For my internal contribution to release 0.3, I had to contribute to the Seneca CDOT Telescope Project. It is a tool for tracking blogs in orbit around Seneca’s open-source community. This blog post will briefly discuss how I tackled this issue and what I learned in the process.

      The Issue Explained

      Issue description

      This issue is a feature request that stated the need for providing logging support to log important events in production as well as help in debugging during development. The issue specified to use Pino for implementing logging support for the project’s backend. Pino is a very low overhead Node.js logger, inspired by Bunyan.

      The Process

      To get started, I first read the contribution guidelines and set up my development environment. Then I read the documentation from the Pino project to get an overview of how to go about implementing this feature. Also, I found an awesome blog on Twilio about Node.js logging. This blog helped me sketch a picture of how I should implement logging.

      I started by installing Pino and Pino-Pretty as a dependency in the project and then created a logger.js module that exports a logger instance which can be used in other modules to implement logging for important events. The properties for the logger instance were configured according to the environment and the log level. The NODE_ENV and LOG_LEVEL environment variables could be set in the .env file. Also, the LOG_FILE environment variable could specify a file path where the logs would be written. When LOG_FILE is not specified, the logs are written to the console using the pino-pretty option.

      For Production Environment:

      If NODE_ENV = Production or NODE_ENV is not set, then the code:

      • Defines a new Logger instance.
      • If LOG_FILE is set, writes logs to a specified file path or outputs them to console.
      • Sets the log level to the LOG_LEVEL environment variable with ‘info’ as the default level.

      For Development Environment:

      If NODE_ENV = Development, then the code:

      • Outputs logs to console (which is by default)
      • Enables prettyPrint option and translates time from epoch time to local time.
      • Sets the log level to the LOG_LEVEL environment variable with ‘debug’ as the default level.

      I also wrote a guide on logging which would serve as a reference for developers looking to implement logging in their modules. It summarizes how to use the logger instance for different environments and some best practices like using a child logger.

      Finally, I opened a pull request for this issue and after going through a thorough review, it was merged into the project.

      The pull request

      Here is an example of how the logger can be used in another module:

      //Import the logger instance from the logger.js module.
      const parentLogger = require('../utils/logger');
      //Create a child logger from the parent logger and pass the module name, so the logs will have more context.
      const log = parentLogger.child({ module: 'module-name' });
      //Start logging events.
      log.info('Important information...');

      Learning Outcomes

      1. Learned how to use Pino to implement logging in a project.
      2. Learned about the best practices of logging in different environments (Production vs Development).
      3. Learned to collaborate with other contributors and get through a rigorous code review :).
      4. Learned how to write good documentation for future contributors.
      5. Learned about the different effects on git history when merging vs rebasing with master branch to resolve conflicts.

      by Hansal Bachkaniwala at November 15, 2019 05:34 PM


      Yiran Zhu

      SPO600 Stage 0.5 – Project Selection

      A week ago, I picked Libav for my SPO600 project selection. Libav is a fork of the FFmpeg project and the Libav project was announced on March 13, 2011 by a group of FFmpeg developers.

      I found the Libav on https://libav.org/ and something about it concerned me right away. It was the News component of the website as it’s last update was on February 12, 2018.

      With this concern, I contacted my course instructor, Chris Tyler, and he suggested that I switch to the original fork, FFmpeg. Originally, I avoided FFmpeg because I thought multiple students couldn’t pick the same library for their project selection but that wasn’t the case.

      In my next blog post, I’ll talk about my journey on building, testing and optimizing FFmpeg, the real Stage 1.

      by naiuhz at November 15, 2019 04:54 PM


      Hansal Bachkaniwala

      Contributing to Jest-Extended Part 2

      For my external contribution for release 0.3, I chose an issue from the Jest-Extended project which I worked on previously. It is a project that provides custom matchers (also called assertions) that can be used in the Jest testing framework. This blog post will briefly discuss how I tackled this issue and what I learned in the process.

      The Issue Explained

      Issue description

      This issue is a feature request that stated the need for a new ‘Array.toSatisfyAny(predicate)’ matcher that can be used in Jest. A matcher is a method/function which compares the expected result with the actual result and returns a boolean (true/false) value. These matchers form the basis of testing frameworks like Jest. The Jest project provides many matchers but users often require new ones to meet project-specific requirements.

      The Process

      To get started, I read the code from the ‘Array.toSatisfyAlll(predicate)’ matcher which is a bit similar to my matcher and followed the steps I would need to create and test a new matcher. They were as follows:

      1. Created a new subdirectory under the matchers directory.
      2. Added the following files to the subdirectory:
      • index.js — An export of the matcher in the format expected by Jest.
      • index.test.js — Test suite that uses the new matcher and makes sure it passes.
      • predicate.js — The function that tests the actual value meets the expected value/behavior.
      • predicate.test.js — Tests for the predicate. Both true/false cases must be covered.

      3. Updated the documentation to include the new matcher.

      4. Added the type definition of the new matcher to types/index.d.ts file.

      After this, I coded the files listed above (using the TypeScript language) and tested them. When all tests passed successfully, the code was committed, pushed and a pull request was created.

      The pull request

      Here is an example of how this matcher can be used in Jest:

      .toSatisfyAny(predicate)

      Use `.toSatisfyAny` when you want to use a custom matcher by supplying a predicate function that returns a `Boolean` for any value in an array.

      test('passes when any value in array passes given predicate', () => {
      const isOdd = el => el % 2 === 1;
      expect([1,2,3,4]).toSatisfyAny(isOdd);
      expect([2,4,6,8]).not.toSatisfyAny(isOdd);
      });

      Learning Outcomes

      1. Wrote custom matchers for Jest.
      2. Wrote documentation for code.
      3. Brushed up on my TypeScript programming skills.
      4. Wrote tests for code.

      by Hansal Bachkaniwala at November 15, 2019 04:53 PM


      Rafi Ungar

      Introducing Comparative Spectrums to the Layer5 Landscape

      During Hacktoberfest 2019, I worked on the website of Layer5, the service mesh community. More specifically, my contributions concerned one specific page of that website: its Service Mesh Landscape page.

      Shortly after making these contributions, I was graciously welcomed into the Layer5 community by its lead developer Lee Calcote, who later directly approached me to assist him in furthering his vision for the Landscape page by implementing “comparative spectrums”. Wishing to focus on my remaining two Hacktoberfest contributions at the time, I suggested surveying the interest of other potential contributors for this project. As such, an informative issue was opened:

      Issue layer5#211: add comparative spectrums to the Landscape page

      A month later, I fortunately found time enough to make some major progress into this issue.


      This issue is quite highly conceptual, much moreso than any other issue I have thus far worked on. I decided that the best way to approach this issue would be to focus on establishing a set of tools that could be used by more-knowledgeable contributors to later construct comparative spectrums—opinionated data visualizations. As such, my first step was to build a set of responsive graphical elements that resembled those provided as examples. These include:

      (1) Sleek information cards:

      or this:

      (2) Gradiented, labelable double-arrow banners:

      (3) Graphically-augmented tables:

      (4) Spectrum-like graphical comparisons:


      (1) Although there may exist libraries that may help construct such graphical elements, I did not want to introduce additional libraries to accomplish my work. Instead, I wanted to make good use of Layer5io’s preexisting Materialize library. Fortunately, Materialize strongly supports the design of card-like structures, allowing me to more easily implement some.

      (2) Unfortunately, it does not seem that Materialize supports the creation of graphical arrows, so I just ended up styling my own.

      (3) These tables can be quite easily implemented and styled with or without the use of <table> tags. I anticipate these will be given a proper fleshing out after the other graphical elements are further developed.

      (4) These unique spectrum graphics present the most challenging, of all the comparison tools, to implement. Snaking spectrum bar aside, I needed to develop a responsive solution that (a) allowed contributors to programmatically place image thumbnails at preset horizontal positions, while (b) ensuring those thumbnails placed at the same position do not overlap with one another (and instead align vertically atop each other).

      This presented a problem to solve: I required a means to ‘float’ elements toward a horizontal line; to ‘stack’ thumbnails atop one another, without overlap, if they are given the same horizontal position. After researching quite deeply into potential solutions to this problem, I ended up stumbling across some CSS property-value pairings that I had never heard of before: display: grid and grid-auto-flow: row dense. Miraculously, I discovered that these could actually function as a basis for a solution.


      After styling the graphical elements, I continued the website’s trend of using Liquid tags (e.g. for) and YAML lists to help modularize my solutions, which should aid future contributors in generalizing them.


      I opened a pull request to share my work as well as invite collaboration:

      Pull request layer5#238: description of tooling introduced
      Pull request layer5#238: example images of tooling introduced

      The pull request was quite well-recieved:

      Comment by Layer5 team lead on pull request layer5#238

      As I mentioned in a recent Layer5 community meeting, I am very excited to continue working with Layer5; to refine and expand upon this set of new, visionary tooling for their landscape!

      by raungar at November 15, 2019 10:43 AM

      Contributing to Telescope 🔭

      The development of Seneca CDOT‘s planet-ending Telescope project is well underway, and I am happy to report on two very different contributions that I made over the course of the last three weeks.

      Kicking off Kubernetes

      Through my Hacktoberfest 2019 work—both with the service mesh community Layer5, specifically, as well as generally with Docker—I first came into contact with the concept of Kubernetes. Since then, I have been interested in learning more about Kubernetes, and was provided an opportunity to do so when I discovered that it was named amongst the MVP features for Telescope.

      I capitalized on this opportunity to ‘open-source’ my research efforts into Kuberbetes by promptly opening up a Telescope issue (#4) dedicated to aggregating Kubernetes-related research and discussion. Since then, I have benefited from learning from those contributing to the thread while aiding in the facilitation and management of its progression.

      I Jest

      As the discussion thread that I started satiated my curiosity about Kubernetes, I decided to branch my efforts out towards a different MVP feature: (unit) testing.

      I discovered that I was not the only one involved in writing Jest tests for Telescope: several other contributors were writing tests. While several of Telescope’s unit testing files were contributed alongside newly-introduced functions, most of the testing files were added separately, as the result of opened issues. At this time, several such issues remain open and without an assignee.

      I wanted to contribute some unit testing to Telescope, but, knowing that many others do as well, I wished to take care to ensure that my work did not render the work of others redudant; that no one else was or will be unknowingly writing the same tests as I was. To do so, I decided to either create a new issue for a yet-untested source code file, or to assign myself to a relevant existing issue.


      While searching for testing-related issues (the type: test label is very useful!), I came across the following issue with no assignees:

      Issue telescope#197: two inactive blog filter functions require unit testing

      This seemed like a perfect opportunity to contribute some needed testing to Telescope (specifically for the check() and update() functions of the recently-merged inactive blog filter).

      Shortly after self-assigning to the issue (assuaging my aforementioned apprehensions), I was well underway to writing tests for the first of the two functions the issue names.

      Syntax and Style

      I am a fan of how Jest encourages documention through the use of common-language structural syntax, e.g. describe(...), it('should...'), expect(...).toBe(...), etc. Wishing for my Jest tests to maintain the same coding style of Lighthouse’s merged and pending test files (but also wishing to utilize the aforementioned syntax), I was a bit dismayed to discover that, across all of the aforementioned files, there exist few structural commonalities. Actually, it dismayed me enough that I opened an issue about it.

      That said, I decided to utilize all of the aforementioned testing methods (and leave the issue of standardizing coding style to future contributors): I used describe() to create a ‘testing suite’ to encapsulate the two tests I wrote for the inactive blog filter‘s check() function. Within the same testing file, I believe that a second testing suite should be added to hold tests for the update() function.

      Pulling it together

      I was perhaps a bit hasty in making a pull request, as its first commit failed our CI testing! I quickly learned to fix my mistakes (through copious use of npm test), and my second commit to the PR passed the previously-failed tests:

      Pull request telescope#211: adds testing for the inactive blog filter‘s check() function

      I’m looking forward to soon having my first contribution to this project reviewed (and merged), as well as continuing to monitor the progress of both of the two discussion threads I’ve opened!

      by raungar at November 15, 2019 08:39 AM


      Calvin Ho

      Release 0.3

      I recall at the beginning of the semester our professor told us sometimes during the course it is ok if we don't have a PR for an external bug as long as we're working on it. Trying to find out what's causing the bug actually takes the longest and the fix could be a change in a line or two of code. As I listened to this story, I thought "Yeah right, what are the chances of this happening to us?"

      One of my goals for this course was to have a bug fix for a project from a big organization. I previously mentioned, I chose to work on a bug from Googly Blockly as it fit the bill, specifically this issue here.

      Blockly is used by developers to create Blockly apps. These apps are usually for students or the public to teach coding in a beginner-friendly way. 

      If the issue with the link didn't provide a good overview here's a quick rundown of the issue. The person filing the bug provided some XML code to load three blocks, one stacked block of two connected blocks and one orphaned block to the right of the stack. The stacked block are pieces that can connect anywhere, whereas the orphaned block is an end piece.

      I should technically be able to:
      1. drag the orphaned block over to the first block in the stack
      2. the orphaned block should replace the second block on the stack
      3. the current second block on the stack should be detached from the stack as it cannot connect to the new second block as it is an end piece
      This is not happening, instead I am able to only connect the orphaned block to the last block in the stack and instead create a 3 stack block.

      I tested each separate block in the stack and can confirm each of them can be connected to the orphaned block individually and this issue is only happening when there is a stack.

      A frequent contributor to the project @BeksOmega gave me some pointers on what to look for as the main culprits:
      • a function called showPreview_ displaying a translucent preview of the result if the user was to connect the block
      • a function called shouldReplace_ should be called when the user drops the block and blocks connect
      • OR none of these functions get called and I have to just trace the call stack and figure out what's wrong
      Guess which route I got? I spent a week tracing the call stack and identifying what each function being called did and returned.

      Following the whole call stack, I finally found the culprit, the following switch statement:

      case Blockly.NEXT_STATEMENT: {  
      // Don't let a block with no next connection bump other blocks out of the
      // stack. But covering up a shadow block or stack of shadow blocks is
      // fine. Similarly, replacing a terminal statement with another terminal
      // statement is allowed.
      if (candidate.isConnected() &&
      !this.sourceBlock_.nextConnection &&
      !candidate.targetBlock().isShadow() &&
      candidate.targetBlock().nextConnection) {
      return false;
      }
      This switch statement was inside the function Blockly.Connection.prototype.isConnectionAllowedThe comment provided in the code perfectly explains why our orphaned end block was unable to replace the last block on the stack.

      I tested a quick and dirty fix of modifying the return false statement to return true to see if it resolved the issue:
      It worked! However, the quick and dirty fix may cause other issues as there was a case specifically written for this scenario, so this change will probably negatively affect other scenarios. I discussed this with @BeksOmega and asked for her opinion raising my concern and pointing out maybe the behavior which she observed that applied to rows was instead the bug.

      A person from Google eventually joined the discussion and we are waiting to hear back from another member until we decide on a way to fix this issue.

      For my second internal issue, I decided to write unit tests for the feedparser function I submitted. At the time of my feedparser PR, the project didn't have any testing packages installed yet. It was fun and different as I have never written tests before, it required us to think of tests for not only normal scenarios, but for edge cases as well. 

      by C3HO (noreply@blogger.com) at November 15, 2019 07:27 AM

      Open Source November Edition

      October, Hacktoberfest is finished and we're moving onwards!

      Our task for November is to experiment with different technologies and revamp the CDOT Planet Feed between the professor's two open source classes. The project is called Telescope and is intended to be written in Javascript for as long as technically possible. My issue for the project this week was to create a parser to parse either a RSS or Atom feed.

      With some discussion and suggestions from @ImmutableBox, @lucacataldo and our professor @humphd. I managed to finally write one with the following

      const parsefeeder = require('parsefeeder-promised')

      module.exports = async function (feed) {
        return parsefeeder.parse(feed);
      };

      Ironically these few lines took a few days. We tried to avoid using the npm package shown and tried to write it from scratch using bent and feedparser. We realized it was kind of impossible to return the data we wanted, and finally resorted to using the feedparser-promised package suggested by @lucacataldo. Feel free to read about the internal discussion of my pull request for the issue here.

      For an external bug as part of release 0.3, I chose to work on Googly Blockly, specifically this issue here.

      by C3HO (noreply@blogger.com) at November 15, 2019 05:41 AM


      Bowei Yao

      Contributing to Mozilla

      I can officially call myself a contributor to Mozilla now. Got my first PR merged here: https://github.com/mozilla/foundation.mozilla.org/pull/3899, and soon, the changes I have made will be reflected at https://foundation.mozilla.org/, accessible to the whole world.

      Aside from the one being merged, there are a few more PRs/issues on the waiting list from me for foundation.mozilla.org, they are:

      There are some back and forth going on with some of these, but I’m sure they can be worked out very soon.

      A couple new things learned from contributing to Mozilla:

      I got docker working with windows… finally. I remember back then, there existed this rather immature relationship between the two siblings who never got along. I still made some sacrifices though – I can’t use oracle virtualbox anymore. But the benefits outweigh the drawbacks: I used to reboot my computer on ubuntu just to use docker – I’m glad I don’t have to do that anymore either. Also, with docker desktop nailed down, I anticipate having an easier time to install other projects on github.

      Foundation.mozilla.org utilizes both django and nodejs. This is very interesting, I’ve never considered using both stacks in the same repository. The entry point is through django (`python manage.py runserver`), and the web server uses the django ORM models to handle database migrations. I could be wrong, but it seems that nodejs is used here solely because of the convenience of npm. The front end uses a wide selection of libraries including babel, react, webpack, eslint, prettier and etc.

      New python libraries learned: pipenv, invoke, and wagtail. I used to use miniconda (anaconda) for package management. Now there’s also pipenv, which can be directly downloaded via pip and used as virtual environment management. Invoke is an interesting library. It wraps around executable python code and changes them into command line invokable tasks. Wagtail is a django-based content management system, which is being used here on certain webpages to create stacks of panels for the front-end.

      The number of repositories under mozilla shocks me: 2.1k and still growing. However, a closer look at the contribution graph filters out a large number of inactive ones. I’m planning on targeting a couple different projects, set them up on my computer (now that I have docker) and see how it goes. Perhaps one of these:

      by Schrodinger's Cat at November 15, 2019 03:22 AM


      Brett Greenham

      AArch64 Optimization – Stage 2 – Optimization Time

      Good news, I’ve gotten access to a much stronger AArch64 box, and I can now test on much more reasonable data. In celebration, I immediately went off and used the power of lossless video to acquire a ~9.3GB video file. And then it was time to run it through csnappy:

      The average time for thing across 100 trials is: 8.372800311 seconds

      Now that’s a lot more reasonable for a test. I’ll get x86_64 data later if it ends up being necessary, but I may not end up making any code changes that affect execution on x86_64, so let’s leave that off for now.

      First trial – compiling with -O3 instead of -O2:

      The average time for thing across 100 trials is: 8.348095289 seconds

      It’s… technically an improvement. But it only saves three hundredths of a second, which makes me think it’s not worth switching to -O3. You might ask ‘why? it’s just changing a compiler flag, might as well leave it’. The answer is that it’s not worth the risk of regressions elsewhere.

      -O3 isn’t just a magic speedup button, it makes changes that have tradeoffs. Generally these tradeoffs are in your favor, but let me present a scenario to you: You’re using a program on an old or cheap processor and you don’t have a lot of CPU cache to spare. Compiling with O2, the program fits in the CPU cache and runs (relatively) well. Then you compile with -O3, hoping for that sweet speedup, and the optimizer does all kinds of cool tricks like loop unrolling that make the executable size larger, but generally result in a speedup. And then your program runs slower, because it no longer fits in the CPU cache. Is this an edge case? Absolutely. But the gain here is so small that it’s really not worth risking things like that.

      Anyway, with -O3 out of the way as a possibility, I’m moving onwards to take a look at some of the preprocessor directives. Previously, I’d noticed things like

      #if defined(__arm__) && !defined(ARCH_ARM_HAVE_UNALIGNED)

      It turns out that this isn’t particularly relevant for our purposes. __arm__ is not defined on AArch64 systems, instead __aarch64__ is. So, let’s take a look around at some other preprocessor directives and see what we can find. The first thing that sticks out to me is this:

       * Separate implementation for x86_64, for speed.  Uses the fact that
       * x86_64 is little endian.
       */
      #if defined(__x86_64__)
      static INLINE int
      FindMatchLength(const char *s1, const char *s2, const char *s2_limit)

      Hmm, that’s interesting. For context, FindMatchLength is called within the main loop of our csnappy_compress_fragment, so if we can optimize that, csnappy_compress_fragment will obviously end up faster. Anyway, this version of FindMatchLength is followed by a different version used for everything other than x86_64. The x86_64 function uses 64 bit loads, and it looks like everything else gets 32 bit loads. I have a feeling I can just chuck ‘|| defined(__aarch64__)’ onto the end of that #if directive and things will work faster on AArch64. Let’s try it:

      The average time for thing across 100 trials is: 8.298365891 seconds

      Hey, that’s definitely faster. Granted, it’s only about a 1%-ish speedup, but it’s a speedup. And this one’s far less likely to cause side effects than jacking up the optimization to -O3, so I’m feeling a lot better about it. All we’re doing is switching from one variation of a function to another variation, and this won’t even affect any platform that isn’t AArch64.

      So, I’ve performed an optimization. Time to send it upstream and call this done? No, not quite. First off, I need to be certain that this change hasn’t broken the library’s functionality. I’m 99% sure that it hasn’t, but good practice demands being 100% sure. I could test that now, and if I suspected that this change may have broken things, I would. But I’m going to leave that for later, and I’ll test functionality at the end (which might backfire on me if I do break something and then have to figure out which change did it, but I figure I’d probably have a pretty good idea).

      As the previous paragraph implied, I’m most likely going to keep looking for some other optimizations to make. I’d like to squeeze at least a few more percent out of this, if I can. I’m not 100% sure the approach I’ll take next, but I have my eye on EmitLiteral (another function called within csnappy_compress_fragment ) for a couple reasons.

      by bgreenham at November 15, 2019 01:27 AM


      Lexis Holman

      Project Update

      Creating a dedicated testing box & some actual code.

      Hey everyone, I have another update on my progress with software portability and optimization, as always, for anyone new to this blog. I am attempting to apply what I have learned so far as a final project and you can check my previous posts on the topic here:

      https://penguins879376500.wordpress.com/category/school-work/project/

       

      New/Old Machine:

      As you were aware I had attempted to build and benchmark my software on a virtual machine, I thought that taking advantage of the tools within VirtualBox such as snapshots and virtual networking that it would make the process go much smoother. In reality however I found that it caused strange fluctuations in output times as well as attempted “optimization” modifications to actually slow the run times down as you will read in a bit.

      Luckily for me I was able to attend a used hardware sale and found myself with a “new” dedicated i5 with a fresh arch install, the process of which will make a good blog post for anyone interested into making the jmp to command line junky. But for now this post will just be about new benchmarks on this new hardware as well as describe some direction as I make a first attempt at optimization.

      Also the networking that I am using for this new machine involves a usb NIC and a virtual installation of pf sense. For anyone who wants to start experimenting with software it is always a good idea to learn a bit about good vm/lab hygiene. Being aware of what machines can see each other or have access to different parts of you internal network are important to know as you move forward. You can check this post for instructions on how to install PFsense onto a Virtualbox VM here:

      https://penguins879376500.wordpress.com/2019/09/14/87/

       

      Before we continue I wanted to say to anyone who has old computing hardware laying around, please do not throw it away, not only is this bad for the environment but cheap or free hardware is how I was able to learn computers; there are many talented people who cannot afford them.

      Please consider donating old computers and hardware, you may just inspire the next generation of tech professionals:

      https://www.rebootcanada.ca/

      https://www.pcnix.ca/donate-your-old-computer/

      (Note: I only found the above examples with a quick google search as an example, it may be a good idea to research further.)

      Now back to the software.

       

      New Build

       Just like in earlier posts, the software was gotten from the github repo:

                      Git Clone https://github.com/Cisco-Talos/clamav-devel

      However this time instead of creating individual build directories with different configurations, we created:

                      Git branch static

                      Git branch vector

      And tailored the source in each branch for the type of build that we are trying to create.

      When it comes to the actual build I have decided on this configuration, as I feel that there is some opporuinity for vectorization within matcher-bm.c, although I am not 100 percent confident with it. This is also the configuration I have decided to settle on while adding my personal modifications to the source code.

      CFLAGS=-ggdb -ftree-vectorize -mavx -fopt-info-vec-missed -fopt-info-vec -fopt-info-vec-note -O2 ./configure –prefix=’/home/<username>/vector_build’ –enable-check –enable-coverage -with-systemdsystemunitdir=no –enable-experimental –enable-clamtop –enable-pcre –disable-llvm

      After we run our configure script we can use the next lines to have our make commands gives us output in regards to decisions made about vectorization in source file’s loops. This information is stored to a file that can be searched for each specific loop with:

                      Make –j<core_count> 2>&1 | tee ~/compiler_output.txt

      Or

                      Make –j<core_count> 2>&1 | grep vector | tee ~/compiler_output_vectorize.txt

       

      To get a more detailed explanation on the actual build process see this link:

      https://penguins879376500.wordpress.com/2019/11/07/project-build/

       

      As well as the provided documentation:

      https://www.clamav.net/documents/clam-antivirus-user-manual

       

      The Test:

      Our test on this new machine is with the exact same data that was used within the virtual environment. However we are now consistently achieving a run time of 2 minutes and 45 seconds after 10 runs with only milliseconds of deviation between them.

      This is in contrast to our previous run times of well over 6 minute, and about 30 seconds of deviation between high and low. You can read about that here: https://penguins879376500.wordpress.com/2019/11/10/main-project-benchmark/

       

      Proper Static build:

      To anyone who was following the blog, or read some of my previous posts. You will have known that I has a bit of an issue with trying to create the software as a static binary. My attempt in doing this was by adding flags to the compiler, where after reviewing the clamav documentation again. I noticed that we were supposed to actually use the –enable-static and –disable-shared flags when calling the configure script. After we did this we were provided with this large map:

      post7

      This output shows a better representation of the number of calls to each function, (because it is using instrumentation instead of sampling.) But aside from a few areas’ we may investigate in the future, it seems our original plan of targeting “cli_bm_scanbuf” in “libclamav” holds true. Let’s look at some code.

       

      Some Code:

      Here is a direct link to each source file of our matcher-bm module:

      Header: https://github.com/Cisco-Talos/clamav-devel/blob/dev/0.103/libclamav/matcher-bm.h

      Source: https://github.com/Cisco-Talos/clamav-devel/blob/dev/0.103/libclamav/matcher-bm.c

      The lines that stand out when inspected/disassembled with perf are:

       

       

      283:  if (p && p->cnt == 1 && p->pattern0 != prefix) 

      Which relates to:

      Movzbl 0x3e(%r13), %edx           ; @ 19.23% usage

       This specific statement is within a large for loop with many conditions between. Not to mention it acts on elements of a c-style linked list that are single entities, in this case one character, and one int. Specifically looking at p->pattern0 != prefix, there is another identical comparison approximately 20 lines of code later, however due to the nature of the linked list pointer potentially being advanced in several places, storing this calculated result is of little value, and after attempting it actually slowed our code (in the virtual environment) by seconds. My thoughts, specifically relating to this would be in the presence of DMA, and if a pattern can be detected; storing these results may be of some worth. We will look into this in the future.

        

      312:  if ((off + p->length > length) || (p->prefix_length > off)) {
                   p = p->next;
                   continue;
             }

                       Which relates to:

      Movzwl 0x38(%r13),%eax            ; @ 12.12% usage

      Again we are looking at values that are resolved using single entities stored as elements of a linked list. The actual reasoning for this, with a lack of comments will have to be researched further; However seems to related to the comparison of signatures. Again determining and storing these values earlier on may relieve some computational overhead.

       

      Loops:

       Loops specifically in this program are handled interestingly, they seem to use for loops with all of the calculation and advancement of pointers being done in the compare and increment part of the loop. Rewriting these to take advantage of SIMD, may provide some optimization however I worry about the length of data being compared, if we advancing a pointer through an array of values, we may need to know the length of that value; as well as requirements involved in forcing alignment of the data structures themselves, again this will require research.

       

                      Example:

           for (; offdata->pos < offdata->cnt && off >= offdata->offtab[offdata->pos]; offdata->pos++)
                      ;

       

       

      Interesting variable:

      On line 249 within cli_bm_scanbuf there is a variable of type uint16_t called idx, I am assuming this is intended to mean index. Where on line 277 we see:

       idx = HASH(buffer[i], buffer[i + 1], buffer[i + 2]);

       

       2 interesting things I have noted are that, HASH seems to report back some kind of index which makes that function an interesting place to investigate for signature lengths. The next and more “suspicious” is that while stepping through this block of code with gdb at –O3, I noticed that when attempting to print this variables value it was reported as “optimized out.” This to me indicated that it may be worthwhile to investigate the intended reason and overall value of this variable.

       

      Summery:

      My intention for this post was mostly to give an update on the change of my dev environment, as well as to keep track as I attempt to cut into the more technical elements of this project. My next post will hopefully include any findings while stepping though individual blocks of code. As well as maybe include some information on how to use gdb to get a better understanding of how a program works.

       

      Love,

      ElliePenguins

      by elliepenguins at November 15, 2019 01:07 AM


      Brett Dennis

      Telescope – Week 3

      This week on our class project was a bit slower for me. With all the discussions and pull requests that I had done and looked at in the previous week, I instead took a step back and primarily looked at what others had submitted in their pull requests this week, and took the time to make sure that a few other pull requests that were just about ready to be merged got the little bit extra work that they needed in order to get into the project. Notably, there seemed to be a few issues with stale reviews preventing users from being able to merge their projects regardless of if they were fixed, and I am unsure if we as a group are aware of dismissing stale reviews when they no longer apply.

      Regardless, I worked on reviewing the inactive blog filter, the fragmentation addition, and commented upon the basic word counter and test.

      by ultimabgd at November 15, 2019 12:39 AM


      Sarika Hanif

      Release 0.3 — Internal Issue

      Release 0.3 — Internal Issue

      For Release 0.3, we are working on modernizing the Plant CDOT feed. The current site looks like this http://zenit.senecac.on.ca/~chris.tyler/planet/. The new project is primarily written in node.js and is now focused on getting key functionality implemented. The pull request that I worked on involved creating the initial config file. After doing some research I found that the config file contains environment variables. I learned that environment variables are important when developing a project, because they allow your app to preform differently based on the environment. Some common places to use the environment variables are for the application host, port, db credentials, etc. When attempting this issue, it was suggested that I use the dotenv module. Dotenv loads environment variables from a .env file into the process.env property (object containing the user environment). This makes using environment variables fairly easy throughout the project.

      After doing some research, I found that I needed to first add the .env file into the .gitignore file, so that sensitive information wasn’t exposed when pushing to github. Then I had to create a template .env file that contained default data for our application. So far, I added NODE_ENV and PORT variables and set them to the default configuration. Whenever a developer wants to run the project, they will copy the template env file and replace all the default values with their credentials and rename the file to .env. From here, I had to include a config file that required and configured dotenv. Whenever environment variables are needed, the config file must be required.

      When I posted my PR, there were really helpful comments. Someone suggested I use .env_example instead of .env_sample, which is better practise. I also got feedback on creating documentation so that developers understand how to use the .env file and using comments to serve as documentation in the .env_example file.

      After making a few changes, it got merged!

      I also made a smaller PR request, because I noticed that when I ran the ESLint test for my initial pull request there were some errors in the dialogue.js file.

      As for discussions within the project, I did some research for sending emails to admin feature and suggested using nodemailer. For release 0.4, I’m thinking of collaborating with the person assigned to this issue. I think I might be a larger feature to implement. I also, reviewed two PRs. The first was to silence npm logs and the other was to integrate the redis url into the .env file.

      So far this project has been fun, I can’t wait to see how it transforms in the next release.

      by Sarika Hanif at November 15, 2019 12:29 AM

      November 14, 2019


      Ilya Zhuravlev

      Setting up the “Telescope”

      In the search for the new Planet

      Caltech Researchers discuss the existence of the ninth planet in our Solar System — and Seneca CDOT is actively looking for a new Planet to become it’s home.

      Greetings everyone!

      I haven’t written posts in a pretty long time, and a lot of things have happened during that time:

      • HacktoberFest2019 finished, all of my PRs got merged (which motivates me a lot!), and my T-Short will arrive at me soon;
      • I have written my mid-term exams — pretty happy with the results;
      • My Major project team and I are working hard towards making our application up and fully working (right now most of the struggles are related to React Context API and Hooks, so I am diving into it);
      • The Telescope project started.

      Telescope

      Photo by Eskipaper

      What is the “Telescope”?
      The Telescope is a new media website for Seneca College’s Open-Source that collects blog posts of all of its members in one place, allowing it’s users to know what happens in the community, what are the projects other members are working on — and allows to manage the Feed formed by those blogposts to be focused on certain needs.

      It is being created by Fall’s 2019 DPS909 (Topics in Open-Source Development) group, which is around 60 programming students.

      THIS IS THE BIGGEST GROUP OF PEOPLE, WORKING ON THE SAME PROJECT, I HAVE EVER BEEN A MEMBER OF!

      But it makes this project even more encouraging and fascinating — because it is OUR project.

      I am not going to cover all of the internal structure of the Telescope project — I will do it in one of the next postings. At the same time, I would like to mention what am I working on.

      Telescope’s Backend has Bull Queues to manage the blog posts — fetch the data with the help of the URL, retrieve the text data from the received response, count the words and calculate reading time, etc.
      In case there is a need to shut down the server, we have to implement the gentle shut down logic that will stop the queues, save all of the processed data and allow the server to continue from the same point it has stopped when turned on. This is what I am working on right now.

      Side Project

      During my Hacktoberfest 2019, I have contributed to the Hexagon-Quilt-Map project, which generates a random blanket from the pieces of various forms (Hexagons, Squares, Triangles). Users can set up the fabrics, size, select the form of the piece — and we are working on implementing more features.

      I find it is a good project for me to practice my JavaScript and React skills.

      As for the upcoming Release 0.4, I am looking for a C++ based open-source project, preferably related to game development.

      https://knitcodemonkey.github.io/hexagon-quilt-map/

      Happy November and Stay Warm!

      My GitHub: https://github.com/izhuravlev

      by Ilya Zhuravlev at November 14, 2019 07:21 PM


      David Medeiros

      Release 0.3 - Failure, Again...

      The old song and dance...

      So we've come to another deadline, and again I am running late. I had an issue to work on, with tania-core and it's SQL to CSV conversion. Turns out this was a much larger undertaking than I expected. I might still work on it for Release 0.4, but I'm unsure if I have the knowledge in SQL and Vue to even attempt tania-box. I worked on it this weekend but with little success, and with time running out and pressures from other courses, I had to find another issue quick. To my surprise, I rediscovered an issue I had commented on but had never been assigned to and jumped on it. I am working on WritingSwarm now, which has already been an interesting project and has opened my mind to a completely different world.

      WritingSwarm and the Issue

      WritingSwarm is a Dapp for writing short stories collaboratively using the Etheresphere Swarm backend. Sounds complicated, and it is, but first what is a Dapp. A Dapp is a distributed app, meaning that it is run on a distributed system. A distributed system is a a network of computers that pass information between them to achieve a common goal. Swarm is a peer-to-peer distributed storage platform and content distribution service. WritingSwarm uses swarm to develop and share short stories. 

      The issue involves adding an auto-save feature to the WritingSwarm site, which is built in Angular. I have used Angular before but never Swarm, so it has been an interesting experience to say the least. Basically, the site already has a save feature, but the user is required to press a save button in order to save their work. My task is to add a toggle switch that toggles between "ON" and "OFF" states for the WritingSwarm environment, and in order to do that I need to use Angular Material, which I have not used before.

      The Environment

      The hardest part of this issue is getting Swarm up and running, because connecting to it is not simple. In order to develop this project, I have to set up a Swarm node, learn Angular Material, and adapt the existing code to provide the auto-save feature. 

      In order to get swarm, I had to use go, which I was fortunate enough to have installed when I was trying to get tania-core up and running. I also had to get geth, which stands for go-ethereum, the official go implementation of Ethereum. Ethereum is basically an open source platform for Dapps. So in order to run WritingSwarm, I had to get an account in Ethereum for go, and download swarm to set up my node. Once I logged in, my node was set up and I was able to serve WritingSwarm on my localhost.

      So it started running, however when I try to add a file, I keep getting an error message. I sent a message to Kortatu, the owner of the repository and I am waiting to hear back.

      UPDATE: Nov 14

      I have not heard back from Kortatu, and the more I hear about Swarm the more I dislike using it. I don't really trust it, and was only using it because I was running out of time. Well now I'm stuck and out of time, so my only option is to find something else. It is very late and I don't expect to do well on this, but I need something to hand in on Friday. I have found a new issue on an old project and will be detailing it in another blog post. I hope to have something working by Friday, and for Release 0.4 I will be more proactive on my external pull request, like I was on my internal one.

      by drwm-base (noreply@blogger.com) at November 14, 2019 06:31 PM


      Yohan Choi

      SPO600: LAB04 - Algorithm Selection Lab

      Introduction

      Music is a large part of our life. In the past, only the privileged group can listen to music to celebrate special events. With the advance of technology, we can listen to music whenever we want. In the past, we can't imagine to make volume down for a music performance. But now we can easily make volume up of music by merely touching the volume button. However, have you thought about how the sound is controlled on a software level? This time, I will investigate how to manage the volume of the music and how it can be effectively optimized. 

      Experiment

      In this experiment, I will scale the 50,000,000 sound samples using a volume factor with a variety of optimization strategies, and I will analyze the results. 

      Test 1

      For the test one, I will use a simple scale function

      static inline int16_t scale_sample(int16_t sample, float volume_factor) {
      return (int16_t) (volume_factor * (float) sample);
      }

      The function simply multiplies the number, but it casts the sample to float.

      O1 option

      result: 94
      real 0m0.588s
      user 0m0.557s
      sys 0m0.030s

      O2 option

      Result: 94
      real 0m0.589s
      user 0m0.566s
      sys 0m0.021s

      O3 option

      Result: 94
      real 0m0.528s
      user 0m0.515s
      sys 0m0.011s

      Analysis

      There is no significant difference between O1 optimization and O2 optimization, but when I applied the O3 option, the difference is substantial. In both real, user, and sys, the performance was enhanced. Just changing the optimization option, brought significant improvement.

      Test 2 - Pre-calculated lookup tables


      for (int i = 0; i < size; i++)
      {
      lookup[i] = scale_sample(value, 0.75);
      value++;
      }

      In test 2, I used a lookup table that stores the value in the range of the sample. However, the process of calculating random values for the samples is not complicated.

      O1 option

      Result: 94
      real 0m0.647s
      user 0m0.625s
      sys 0m0.020s

      O2 option

      Result: 94
      real 0m0.645s
      user 0m0.622s
      sys 0m0.021s

      O3 option

      Result: 94
      real 0m0.644s
      user 0m0.642s
      sys 0m0.000s

      Analysis

      As expected, there is no improvement in performance. Besides, the time to calculate the lookup table makes the performance is slower than other versions. After changing the optimization options, there is no striking change. If a process to get value is involved, and the data size is enormous, the lookup strategy will be useful. 

      Test 3- Fied point integer


      static inline int scale_sample(int sample, int volume_factor)
      {
      return ((volume_factor * sample) >> SHIFT);
      }

      For Test 3, I used a fixed point integer. It makes float calculation to integer calculation, which is easier for CPU. O1 option.

      Result: 873real 0m0.555suser 0m0.523ssys 0m0.030s

      O2 option

      Result: 873
      real 0m0.553s
      user 0m0.531s
      sys 0m0.020s

      O3 option

      Result: 873
      real 0m0.524s
      user 0m0.483s
      sys 0m0.039s

      Analysis

      Compared to the previous strategies, the performance is better. However, there is no significant difference with test 1 when I applied O3. If the calculation on float point becomes more laborious, there will be benefits. I'm not sure about it, but I guess the O3 option may apply a fixed point calculation. 

      Conclusion

      In this lab, I had time to see different strategies to optimize volume control. If the process of getting data is laborious, and the size of data is significant, using a lookup table will be a good strategy. Still, in this case, since the calculation is not that complex, it makes the run time of the application slower. In the case of fixed, without O3 optimization, it represents the highest performance. Given the process of the application, we can apply a variety of strategies to optimize the application. 



      by Yohan (noreply@blogger.com) at November 14, 2019 07:24 AM

      November 13, 2019


      Cindy Le

      OSD 600 Release 0.3 – Create-React-App

      In my Release 0.2, I made an attempt to resolve dead links found in the create-react-app repository but unfortunately, this issue couldn’t be resolved by someone outside Facebook since the new shortlinks needed Facebook Open Source to set up the DNS for it. This issue won’t be resolved for a while because there are actually a lot of deadlinks in their files.

      My PR was added to their project board so they don’t forget to fix it in v3.3

      After I graduate, I want to get a job as a front-end/full stack developer so working to fix an issue in the create-react-app repo would look great on my resume. I had some trouble finding a good issue for my Release 0.3 because they were either out of my scope or already taken. I eventually found one that has been hiding and no one really paid attention to. Sure, it was originally opened September 20 and someone wanted to do it on October 8th but that person asked “how do I get started?” and that was the end of that conversation. The developers recently added a new logo (changed the colour from blue/white to green/grey) and they wanted a branding page.

      My PR addressed their new colour scheme. However, I am aware that some people rejected the new logo and colour scheme as it kinda messed with the projects already made in React using the previous stuff so there was a merge to revert back to the old logo. I didn’t follow too closely to what was happening with that since their website is currently using the new logo and colours so I just went ahead and submitted my PR with the Logo Resources page. They use docusaurus which I find to be very organized so maybe in Release 0.4, I will suggest this to be used in the Telescope project.

      by cindyledev at November 13, 2019 02:14 AM

      November 12, 2019


      Calvin Ho

      SPO600 Project Selection

      In an attempt to learn more about the backend we're using as part of our OSD600 internal project and to make contributions to a widely used open source project, I decided to optimize Redis as my SPO600 project.

      First I had to create some data:

      for ((x=0; x<100; x++)) ; do echo "$(dd if=/dev/urandom | tr -d -c "a-z" | dd bs=15 count=1 2>/dev/null)=$(dd if=/dev/urandom | tr -d -c "a-z" | dd bs=10 count=1 2>/dev/null)" | cat >> testData ; done

      this block of of code will run a for loop 100 times
      within the loop:
      • output to the screen randomly generated key(15 chars) value(10 chars) pairs made up of lower case letters
      • .take whatever is outputted on the screen and append to a file called testData
      At the moment I can run a perf record with the following bash line

      perf record redis-cli set foo bar
      • this allows me to do a sampling of the command set foo far
      which generates the following report:


      Upon further research, Redis actually comes with its own benchmarking tool called redis-benchmark. A great source of documentation published on DigitalOcean can be found here.

      An example of a command I ran:

      redis-benchmark -t set -n 10000

      A simple breakdown of the command
      • redis-benchmark - the command
      • -t (command)- use a subset of tests specified by command, in this case I'm using set
      • -n (number of requests)- number of commands to carry out 

      The tool is useful because it allows me to track how many seconds it took to complete each operation for x amount of requests, which in this case it to set 10000 key value pairs. 

      The command also allows users to specify how big each randomly generated block size (for the key/value pairs) should be(default 3), to ensure consistency between operations. I also am not only limited to just benchmarking the set only, I can specify in the CLI set,get and redis-benchmark will then also benchmark how long it took to get x amount of values.





      by C3HO (noreply@blogger.com) at November 12, 2019 10:33 PM