Planet INdT

Ian Lawrence: Microsoft Education Delivery

2015-08-07T17:06:43+00:00

We spent last year designing and creating MED so it was nice to find this video created by Microsoft explaining what it is and how it works

Francisco Alecrim: SCRUM primeiros passos

Francisco Keppler Silva Alecrim — 2015-05-21T04:24:28+00:00

Esta semana recebi o e-mail de um amigo que está tentando implantar SCRUM no time que está liderando. Quando trabalhavamos juntos na BRISA tecnologia, fiz uma apresentação sobre o assunto e tentei implantar SCRUM em um dos times. Abaixo seguem alguns links e apresentações que recomendo.

O link abaixo leva para a apresentação que modifiquei para o time do MegaRushing.

Introdução SCRUM – modificado por Alecrim [ PDF ] [ ODP ]

Ian Lawrence: Jozian Ltd

2015-04-06T14:35:34+00:00

Needed to make a Company Presentation recently so thought I would add it here.

Bruno Araujo: Unity3D: Typing effect with sound for Text UI components

2015-02-11T18:24:04+00:00

Learning and understanding the concepts of Unity3D is no easy feat, as anyone starting to use the engine can attest, but at the same time is a great experience not only in the programming side but also in the game design one. In the spirit of giving back to the awesome game development community and also to the Unity3D community, I’ll be sharing some insights and techniques that I’ve learned in the process.

One of the results of this process is a demo with some UI features. You can see the demo in action here (Windows only), and there is a (rather heavy) GIF here, if you can’t use Unity WebPlayer for any reason.

There are three things happening in the demo: transitions between three Canvas components, the text typing effect with sound and the rotating background. For this post, I’ll be discussing the second feature; links for posts discussing the other topics will be at the end of the post.

For making the typing effect, you just have to display the desired text letter by letter, with a small delay, as they would appear if a person were typing them. But first things first: create an Text component in the hierarchy and give it a name; in this example the Canvas will be called IntroCanvas and the Text will be called LeftTitle. As in the previous post, create a script and add it under a GameObject, usually an empty one, then save in it a reference to IntroCanvas and to the Text component.

using UnityEngine;
using UnityEngine.UI;
using System.Collections;

public class Director : MonoBehaviour {
    private GameObject introCanvas;
    private Text leftTitle;

    void Awake() {
        introCanvas = GameObject.Find("IntroCanvas");
        leftTitle = introCanvas.transform.Find("LeftTitle").
            GetComponent<Text>();
    }
}

The function below receives a Text component and gradually fills it with the desired text, blinking a simulated cursor a couple times. The speed parameter is the delay between letters in seconds. In a nutshell, this function simulates a blinking cursor twice and then proceeds to write the text.

IEnumerator writeText(Text obj, string text, float speed) {
    string gradualText = "";
    obj.text = "_";
    yield return new WaitForSeconds(0.2f);
    obj.text = " ";
    yield return new WaitForSeconds(0.2f);
    obj.text = "_";
    yield return new WaitForSeconds(0.2f);
    obj.text = " ";
    yield return new WaitForSeconds(0.2f);
    obj.text = "";
    for (int i = 0; i < text.Length; i++) {
        gradualText += text[i];
        obj.text = gradualText + "_";
        yield return new WaitForSeconds(speed);
    }
    obj.text = text;
}

An usage example of this function:

public void animateLeftTitle(){
    StartCoroutine(writeText(left, "main title", 0.25f));
}

For a discussion about StartCoroutine(), you can take a look at the official Unity Manual about it. In a few words this allows functions to be executed over many frames, instead of having to finish entirely on the current frame, like Update() does. The yield statement tells the function to stop and resume in the next frame.

Now, let’s add some sounds! For each letter we’ll playing a short “switch” sound, randomly chosen from a pool for giving the illusion of pressing keys in a real keyboard. I’ve used a set of sound assetrs that I already had (you can get them and a lot of other assets here). For this task, as done with the Director, create an empty object and attach a script to it; I’ll call it TypingSoundPlayer.cs, and call the object TypingAudioSource.

using UnityEngine;
using System.Collections;

public class TypingSoundPlayer : MonoBehaviour {

    private AudioClip[] typingSounds;
    private AudioSource audioSource;
    private int size;

    void Awake () {
        int i = 0;
        audioSource = GameObject.Find("TypingAudioSource").GetComponent<AudioSource>();
        Object[] objSounds = Resources.LoadAll("Audio");
        size = objSounds.Length;
        typingSounds = new AudioClip[size];
        foreach (Object obj in objSounds)
        {
            typingSounds[i] = obj as AudioClip;
            i++;
        }
    }

    public IEnumerator playTypingSound()
    {
        audioSource.PlayOneShot(typingSounds[Random.Range(0, size - 1)]);
        yield return null;
    }

}

This script will load all sounds and put them in an array at the Awake() function, and now the writeText() function just have to call playTypingSound() for playing a random sound from the array. The modified Director class can be seen below:

using UnityEngine;
using UnityEngine.UI;
using System.Collections;

public class Director : MonoBehaviour {

    private GameObject introCanvas;
    private Text leftTitle;
    private TypingSoundPlayer typingSound;

    void Awake() {
        introCanvas = GameObject.Find("IntroCanvas");
        typingSound = GameObject.Find("TypingAudioSource").GetComponent<TypingSoundPlayer>();
    }

    IEnumerator writeText(Text obj, string text, float speed) {
        string gradualText = "";
        obj.text = "_";
        yield return new WaitForSeconds(0.2f);
        obj.text = " ";
        yield return new WaitForSeconds(0.2f);
        obj.text = "_";
        yield return new WaitForSeconds(0.2f);
        obj.text = " ";
        yield return new WaitForSeconds(0.2f);
        obj.text = "";
        for (int i = 0; i < text.Length; i++) {
            gradualText += text[i];
            obj.text = gradualText + "_";
            StartCoroutine(typingSound.playTypingSound());
            yield return new WaitForSeconds(speed);
        }
        obj.text = text;
    }

Now after each text update typingSound.playTypingSound() is called, giving the desired sound effect.

You can go to the other two related posts: using multiple canvases or make a rotating background (soon).

Bruno Araujo: Unity3D: Using multiple Canvas components for switching UI components on the fly

2015-02-09T05:38:10+00:00

There are three things happening in the demo: transitions between three Canvas components, the rotating background and the text typing effect with sound. For this post, I’ll be discussing only the first feature; links for posts discussing the other topics will be at the end of the post.

In this post I’ll demonstrate how to use more than one Canvas when using Unity UI, allowing the screen to show different UI elements at different moments. Probably someone already devised a better solution, but this one proved to be relatively simple and easy to implement. You’ll need Unity3D 4.6 or newer for this task, as the new UI was introduced at this version.

So, in this experiment, I’ve created three Canvases to hold the UI elements needed:

IntroCanvas, for showing text corresponding to the main and secondary titles of the game, along with a button;
StoryCanvas, also displays texts and a button in different positions;
GameCanvas, which displays a button for resetting the game.

The TranslucentScreen objects are just empty Image components with RGBA values of (0, 0, 0, 170), which gives a dark translucent effect to the canvas.

The key for using these canvases without one overlapping each other is only enabling the canvas which will be used at the moment; this is possible because all of them inherit from GameObject, and thus inherit the relevant SetActive() function.

First, create a empty GameObject in the hierarchy, and under it create a new script to hold the transitioning logic; it will be called Director.cs in this example. This script will hold references to the Canvases and active only the first one (IntroCanvas):

using UnityEngine;
using UnityEngine.UI;
using System.Collections;

public class Director : MonoBehaviour {
    private GameObject introCanvas, gameCanvas, storyCanvas;

    void Awake() {
        introCanvas = GameObject.Find("IntroCanvas");
        storyCanvas = GameObject.Find("StoryCanvas");
        gameCanvas = GameObject.Find("GameCanvas");
    }

    void Start()
    {
        introCanvas.SetActive(true);
        storyCanvas.SetActive(false);
        gameCanvas.SetActive(false);
    }
}

When the scene loads, IntroCanvas will be active and will react to user input. When the user clicks on the “continue” button, the UI must change from IntroCanvas to StoryCanvas; this can be done by simply disabling one and enabling the other.

For an explanation on how to connect the button click event and the function below, you can refer to the video tutorial available here, around the 8:20 mark.

So, for this task, you can just call SetActive() directly and call it a day:

public void switchToStory(){
    introCanvas.SetActive(false);
    storyCanvas.SetActive(true);
}

But what if you want a transitioning effect? A simple and nice effect (IMHO) is fading in and out, which can be accomplished by adjusting the alpha value of the entire canvas. For this, add a CanvasGroup to each Canvas. This component has a convenient alpha variable that will be used for the fading effect, demonstrated in the following functions:

IEnumerator fadeIn(GameObject obj, float speed) {
    float increment;
    obj.SetActive(true);
    CanvasGroup cv = obj.GetComponent<CanvasGroup>();
    while (cv.alpha < 1) {
        increment = speed * Time.deltaTime;
        if (cv.alpha + increment > 1) cv.alpha = 1;
        else cv.alpha += speed * Time.deltaTime;
        yield return null;
    }
}

IEnumerator fadeOut(GameObject obj, float speed) {
    float increment;
    CanvasGroup cv = obj.GetComponent<CanvasGroup>();
    while (cv.alpha > 0) {
        increment = speed * Time.deltaTime;
        if (cv.alpha - increment < 0) cv.alpha = 0;
        else cv.alpha -= speed * Time.deltaTime;
        yield return null;
    }
    obj.SetActive(false);
}

fadeIn() will activate the object and increment the alpha value until 1, and fadeOut() will decrement it until 0 and disable the object. Just pass the Canvas object to these functions and a fading speed, and all is set. So, we can make the transition like this:

public void fadeToStory() {
    StartCoroutine(fadeOut(introCanvas, 2.0f));
    StartCoroutine(fadeIn(storyCanvas, 2.0f));
}

In two seconds IntroCanvas will fade out and StoryCanvas will fade in. The same effect is used when transitioning from StoryCanvas to GameCanvas and from GameCanvas back to IntroCanvas.

You can go to the other two related posts: make a typing effect for text with sound or make a rotating background (soon).

Danilo Cesar: A brief discussion about Crosswalk’s JS, JAVA and Native Extensions performance

2015-02-05T15:06:07+00:00

I had a lot of fun recently playing with Crosswalk. I was basically testing it’s features and extensions when it hit me: I know it’s possible to write Native extensions for crosswalk Tizen, but is it possible to write Native Extensions for Crosswalk-android? How would it behave?

Crosswalk android already provides a way to load Java extensions, and the Android-NDK provides a way to call a native library from a Java method. We just have to glue those things together.

Disclaimer:
I won’t talk here about how to build a library with NDK or how to write a regular extension with Crosswalk, as those topics are already extensively covered over the Internet. I assume you already have an library.so and a working crosswalk-android extension from the Tutorials.

Adding the .so inside your apk

Crosswalk-samples includes an example application named “extensions-android“. This application builds the extension code first using an “ant” recipe and then builds the html application itself and put both inside an apk file.

By the end of the extension build process, the ant recipe opens the extension.jar file to include the META-INF information inside that file. It’s the perfect time to inject our native library inside the package.

We only have to change the build.xml recipe to include the library.

<copy file="libs/armeabi/libMyLibrary.so" todir="${build}/lib/armeabi-v7a/" />
Be sure to do it between <unjar> and <jar> as we’re taking advantage of the fact that the extension jar is being opened and closed to include the META-INF.

Performance measurement

To make a performance test I used an algorithm to find the nth prime number. The approach is the simplest (no pre-built arrays, no recursion) and dumbest (no short-cuts, no AKS, etc) one, and share the same implementation. The implementation is the following:


function findPrime(n)
{
    var i, count = 0;
    for (i = 1; count < n; i++) {
        if (isPrime(i))
            count++;

        if (count == n)
            break;
    }   
    return i;
}

function isPrime(n)
{
    var i = 0;
    if (n == 1 || n == 2)
        return true;

    for (i = 2; i < n; i++) {
        if (n % i == 0) {
            break;
        }   
    }   

    return (n == i); 
}

_{Converting it to JAVA or C would be ridiculous easy.}

Performance Results

I first tried to find the 10th prime number and the result is the following:

This was quite expected, right? I mean, the “context switch” between JS->JAVA or JS->JAVA->C has a price to be paid, there’s the serialization/deserialization process that takes some time to finish, and since the core operation is not really time consuming (a short iteration between 1 and 29, checking if they’re prime numbers) I can imagine why JAVA and C took more time. Let’s try again and find the 100th prime:

OK, It’s a bit odd, the loading time is different from execution to execution and it affects the first time the code is executed. Ignoring the higher (usually the first execution) and lower values (got lucky?) give us a more sane result, but still… What about the 10Kth prime number?

It doesn’t make any sense, JS is faster?! What’s going on?

Divisions/Modulus are expensive. Even if I build the Native Library with armeabi-v7a it gets 30% better and goes bellow 10 seconds, almost the same performance accomplished by JAVA but still worse than Javascript.

It happens that Modulus operation can be improved, and I believe that the V8 is smart enough to use some of those techniques. So, looking for primes might not be a fair way to measure performance.

Performance Results, round two

A second approach would be to avoid divisions and use only plain additions. Maybe just sum the sequence 1+2..+N, repeating it N times?
Maybe 1-2+3-4…N to avoid overflow?


function sum(n) {
    var signal;
    var i, j;
    var total = 0;
    var r1;
    var start = n / 2;

    for (j = 0; j <= n; j++) {
        r1 = start;
        signal = -1;
        for (i = 0; i<= n; i++) {
            r1 = r1 + (i * signal);
            signal = -signal;
        }
        total = total + r1;
    }
    return total;
}

_{As the final value is -n/2, I’m starting the sum with n/2 to get 0 as the result.}

For a small sum of ten elements:

Makes sense, right? The context switch costs more than the sum of the 10 elements directly on javascript. Let’s see the 50K results:

As you can see here, without space for smart optimizations on the JS side, the native results are 3x better than the javascript. Nice!

Conclusion

There’re a few lessons we learned here:

1 – The cost of going from JS->JAVA or JS->JAVA->C cannot be ignored. It was a bit less than 1ms (from the 10 additions chart). Think about that before putting a native call inside a huge loop.
2 – Consider measuring your optimizations before accepting it in your source code. Sometimes things are not obvious.

Ademar de Souza Reis Jr.: Retrospectiva 2014

2015-01-13T02:53:38+00:00

Um pouco atrasada como de costume, mas vamos à nossa retrospectiva, tradição desde 2006! :-)

Acho que 2014 foi o ano da consolidação dessa nossa fase “família” em Canela. Foi um ano mais calmo, sem grandes mudanças. Mesmo assim tivemos alguns destaques: nossas férias de Fevereiro/Março nos EUA, nossa viagem de férias pra Curitiba em Novembro e como não poderia deixar de ser, as constantes novidades no desenvolvimento da Sofia.

Férias 2014 – Inverno nos EUA (Fevereiro/Março)

Uma boa viagem de férias estava nos planos há algum tempo e como quem nos conhece sabe, nossa ideia de férias ideais envolve frio, neve e um longo tempo fora de casa (relativamente falando). Após algumas análises de custos, benefícios e datas, resolvemos tirar férias nos EUA no final do inverno. Nossa viagem durou um total de 26 dias, passando por Westford/Nashua (próximo a Boston), Sugar Loaf Mountain (Estação de Esqui no Maine) e Nova Iorque. Só não deu pra ir até o Canadá pois os passaportes não saíram a tempo pra solicitar o visto. :-(

A Sofia tinha então 1 ano e 10 meses e isso representou alguns desafios e mudanças no nosso ritmo de viagem, mas de modo algum nos impediu de aproveitar bem cada momento. E pra ela foi o maior barato: mesmo bem novinha, ela brincou muito e hoje ainda se empolga quando vê as fotos.

Viagem EUA 2014 (clique para ampliar)

Nova Iorque (NYC)

Chegamos e saímos dos EUA pelo aeroporto de Newark (Nova Jersey, vizinha de NYC). Na chegada (final de Fevereiro) ainda havia bastante neve e gelo acumulados e na volta (final de Março), era raro encontrar gelo, mas pegamos uma frente fria com vento e temperaturas de até -8C durante o dia. O negócio foi se agasalhar bem e carregar a Sofia bem embrulhadinha no carrinho.

Em NYC fizemos o programa padrão de qualquer turista, mas num ritmo um pouco mais lento: Liberty Park, Manhattan, 5a Avenida, Empire State, Central Park, Museu de História Natural, Estátua da Liberdade, Times Square, etc. Foi a minha segunda vez em NYC e a primeira da Viviane e da Sofia.

No Museu de História Natural – NYC

Westford – MA / Nashua – NH

Saindo de NYC de carro, fomos pra Westford, cidade próxima a Boston, no estado de Massachusetts. É lá onde fica um dos principais escritórios da Red Hat (empresa onde trabalho) e como eu trabalho de casa, aproveitei pra visitar alguns amigos e passar alguns dias trabalhando no escritório, enquanto a Viviane ficava passeando com a Sofia — geralmente em Shoppings e Outlets.

Comida típica :-)

O inverno naquela região é bastante rigoroso e encaramos temperaturas de até -30C em alguns dias. Mas como eu costumo dizer, lá se passa menos frio do que aqui no Brasil, pois eles tem aquecimento em todo lugar. Mancada foi deixar algumas compras no carro durante uma das noites: pela manhã estava tudo super congelado. Brincamos um pouco na neve por lá, só pra dar uma aquecida pois a brincadeira de verdade ainda estava por vir…

SugarLoaf – Maine

Eu disse que a gente gosta de frio, não é? Essa provavelmente foi a melhor parte da viagem: aproveitando o convite de uma colega de trabalho, fomos pro extremo norte/nordeste dos EUA, no estado de Maine, bem próximos da divisa com o Canadá. Lá existe uma famosa estação de Esqui chamada SugarLoaf que é muito legal. E o que não faltou foi neve. Ainda mais interessante é que a temperatura estava relativamente amena: não baixou de -10C, o que permitiu que aproveitássemos bastante nosso tempo por lá.

Snowshoeing

Revendo Amigos em Curitiba (Novembro)

Curitiba é a cidade onde morei de 1998 a 2007 (com um intervalo de 1 ano em Manaus) e foi lá onde eu conheci a Viviane e nos casamos. Sendo assim, temos muitos amigos por lá. Já faziam 7 anos que não aparecíamos então resolvemos fazer uma viagem rápida, com a principal intenção de rever os amigos.

Foi uma viagem tranquila, sem muito planejamento ou stress. Fomos de carro (cerca de 1500 km ida + volta) e permanecemos lá por 13 dias. O suficiente pra visitar vários amigos (vários deles hoje com filhos), comer barreado em Morretes, pastel na feirinha do Largo da Ordem, ir brincar com a Sofia no Parque Barigui e como todo Curitibano, “passear no shopping” (incluindo ver Interestelar no IMAX).

Jogo de tabuleiro na casa do Milton, com Thiago e Danilo.

O que faltou foi acampar no Parque Estadual do Pico do Marumbi. Eu tinha tudo planejado pra passar uma noite acampado ao pé da Montanha com Viviane e Sofia, mas descobri na última hora que a área de camping do parque foi fechada há 2 anos e tive que abortar os planos. :-(

Vida em Canela e desenvolvimento da Sofia

Continuamos muito satisfeitos com nossa escolha de morar aqui no interior. Acho que 2014 foi o ano da consolidação da vida “família” por aqui. Terminamos a extensão da casa, eu assei várias carnes agora que tenho uma churrasqueira e um kit de churrasco (incluindo uma faca de Gaúcho tchê!), a Viviane plantou várias coisas no jardim e tem se divertido cuidando dele e a Sofia adora brincar no quintal e observar os pássaros que vem nos visitar com frequência (o que deu pra ver esse ano: beija-flores, pardais, pica-paus, periquitos, tucanos e papagaios, entre outros).

Vida em Canela – 2014 (clique para ampliar)

Falando em Sofia, ela fechou o ano com 2 anos e 7 meses e nós estamos muito satisfeitos com o desenvolvimento dela: ela já não usa fraldas, está super tagarela (a brincadeira favorita dela é colocar dois objetos pra “conversar”), entende relativamente bem conversas em Inglês, conta e identifica algarismos e quantidades até 10 em Português e Inglês, identifica e ordena as letras do seu nome, come e bebe sozinha (aliás, nunca foi de usar chupeta nem mamadeira), sabe bem o que é pequeno/grande, longe/perto, encima/embaixo, subir/descer, aberto/fechado, claro/escuro, cores (tudo em Português e Inglês) e é bem independente de modo geral. Ela está na fase dos “porquês” e vive nos surpreendendo com frases cada vez mais complexas e imaginativas, o que tem sido o maior barato. O que não foi tão interessante assim foi ter que assistir Frozen umas 35 vezes e ouvir “Let it Go” durante boa parte da viagem pra Curitiba, mas felizmente acho que o pior já passou. :-)

Sofia em 2014 (clique para ampliar)

Ainda falta explorar um pouco mais a região de Canela: até que fiz boas trilhas de Mountain Bike, mas não foi nesse ano que eu desci o Rio Parnaíba de rafting e ainda temos vários pontos turísticos a visitar. Coisas pra 2015.

Mas e a lista de 2014? Bem, fiquei no 50/50:

OK! Fazer pelo menos uma longa viagem de férias com Viviane e Sofia
FAIL! Visitar um país que eu ainda não conheça
OK! Terminar as obras da casa
FAIL! Acampar pelo menos 1 noite com Viviane e Sofia
FAIL! Descer o Rio Paranhana de rafting
OK! Organizar melhor minhas fotos e vídeos, incluindo backup online

Ah, e a Sofia pediu tanto (NOT), que eu tive que comprar um PS4. Mas o irmãozinho ou irmãzinha que ela também está pedindo ficou pra mais pra frente. ;-)

Não tenho uma lista de objetivos pra 2015, vamos ver o que a vida nos reserva. O único plano é fazer mais uma viagem de família, quem sabe novamente pra Patagônia no inverno aqui do hemisfério Sul.

É isso aí, um feliz 2015 a todos e até a retrospectiva do ano que vem!

Link Permanente | Nenhum comentário
Post tags: retrospectiva

Bruno Araujo: Porfolio status: ONLINE

2014-11-08T04:00:42+00:00

So, I’ve just created a portfolio page for listing some things I did, just click the link named “Portfolio” (duh) at the top of the page. Eventually new stuff will be added, and I’ll try to keep it as updated as possible.

Ian Lawrence: Making Magic

2014-08-20T16:26:55+00:00

South Sudan

Is a complicated place. Statistics on the country are shocking:

One out of every seven children dies before their fifth birthday
One of the worst maternal mortality rates in the world
One out of seven women who become pregnant will probably die from pregnancy-related causes
More than half of children between the ages of six and 13 are not in formal education
84% of women cannot read or write. Only 6% of girls who start school ever finish

There is also of course an ongoing conflict with warlords, armed rebellions and clashes over cattle. Given this backdrop very few people here in South Sudan have hopes or fears which are entirely divorced from this conflict.

The fighting has been going on for so long that the root causes of the violence are difficult to know. South Sudan was established as a country in 2011 following a 20 year civil war with northern Sudan. This war was largely a religious and ethnic conflict which often descended into genocide - most famously in the Darfur region. Millions of civilians were killed.

Three years ago when South Sudan finally achieved it's independence from the north there was a great deal of optimism. But late last year a political battle between President Salva Kiir and Vice President Riek Machar ignited a new civil war. Because the two men were from different tribes the fighting has once again broken out along ethnic lines. Ethnic war is an an especially deadly sort of conflict because it can easily spill over into civilian populations.

Fighting in the new civil war is largely between the Dinka and the Nuer - South Sudan's two dominant tribes. When fighting broke out the Nuer stormed the gates of the UN to escape an unfolding massacre at the hands of Dinka fighters. Over the course of a few days thousands of Nuer were gunned down in the capital city where they represented a significant minority. In other parts of the country however Dinka were killed in equal measure in heavily Nuer regions.

South Sudan is of course also a country filled with millions of civilians who are desperately and with the greatest difficulty trying to transcend this history and establish a society based on democratic and egalitarian ideals. Burdened by decades of resentment, revenge and almost ceaseless fighting this is proving to be extremely difficult. I see it as my obligation to support them with this if I can and supporting them means giving them the correct items like seeds, fishing kits and fuel efficient stoves they need at the correct time to start to begin rebuilding their lives.

What is MAGIC

Maybe it helps to think of aid distribution as a sort of e-commerce system where aid items are 'products'. Instead of a 'customer' choosing which items to add into their cart we have to make a decision of who gets what based on the results of a countrywide Food Security Assessment (which classifies areas as either Stressed, Crisis, Emergency or Famine). Conceptually then we can create a shopping cart of items for each payam (a payam in South Sudan is like a District in the UK) and pass this info to other organizational units like LOA (Letters of Agreements or Contracts with Suppliers or NGO's), Procurement, Logistics, Distribution and so on.

Each unit can modify the cart either by updating its contents or changing its geographical location (in a warehouse, with an NGO partner etc). At any point we can establish where the 'shopping cart' of items is and we can generate reports and statistics about its passage through the MAGIC system all the way from food assessment through planning to doorstep delivery

One clear advantage to building the system this way is that we now can establish a link between 'donors' on one hand and 'recipients' on the other. To continue the e-commerce analogy without MAGIC donors would pay for a shopping cart of items and they would not know to whom these items had been delivered. With most L3 donations running into the millions of dollars this is a problem. A second advantage is more subtle but might be useful if we want to try to innovate on aid at some point in the future. If we wanted for example to allow donations not only from Governments but wanted to accept donations from anybody - perhaps raising money with Bitcoin then we could generate a report for each bitcoin donor and add that to the blockchain for historical record

Before all this though the system needs to work and be tested in the field. That is what I have been doing these last two weeks and it was great fun!

Ian Lawrence: A Bitcoin project

2014-03-26T21:10:58+00:00

Some words about hedging

The Bitcoin market is pretty fresh and unstable at the moment. It is greatly affected by any news on this market. We expect slow grow of exchange rates in the long-term. This assumptions is based on the following facts:

The world requires anonymous currency. It is used not only for illegal transactions (like buying/selling drugs, weapons etc.), but also as a currency which is free from any political influence, not affiliated with any country, etc.
Close-to-zero transaction fees. If an operation is made within the Bitcoin infrastructure you have no additional fees for running a bitcoin account
Deflationary nature of bitcoin. The emission amount is fixed and could not be extended. Since (like any other currency) bitcoin's will be lost - the price of the rest should increase to compensate that

Unfortunately, all other *coins (Litecoin, Dogecoin, etc.) have close-to-zero influence on bitcoin exchange rates, while bitcoin exchange rates affects this currencies greatly. Detailed analytics of exchange rates dynamics on 3 different exchange markets revealed 0.00037% probability of getting profit by running exchange operations between different *coins (mostly due to exchange fees of sites)

Our opinion is that it is a nice time to start getting bitcoins :) It's exchange rates have still not recovered after the MtGOX failure, but recent research revealeds that MtGOX may have money on it's account, so we expect constant growing exchange rate of Bitcoin in next 3 months

What needs to be implemented?

This section describes a scope of the project. The main target of current project is to provide an easy way to perform easy transactions within system. So we need just three points of functionality:

Adding money to amount (cash-in)
Transferring money to other person's account
Cash-out

Additionally to simplify people's life transferring money to other person's account should track "friends list" to simplify multiple transactions to other persons.

Since world is mobile now - all that operations should be available both from web-interface and mobile apps.

What will be great to have in future?

This section describes functionality outside initial project scope, but could be considered in future.

"Send me money" button. When user click's it a popup appears on site where person could send money in one click
QR-send. Using mobile you could scan a QR-code and confirm sending money in one click

What about security?

Security is a main concern of the project. Since we're building a system on top of bitcoin - there is no way to reverse any transaction.

So let's review possible approaches.

(rejected) Single bitcoin account managing all users accounts

We have just one bitcoin wallet, which contains all coins owned by our users. All in-system transactions are tracked by our system, same applies to account balances.

Pros: + Very easy to implement + Extremely fast (since there are no "real" transactions when we pass money from one person to other)

Cons: + extremely risky. Let's imagine situation when hacker was able to access account. He will be able to move all bitcoins to other address resulting in system crash

(rejected) Separate bitcoin wallet for each user

Each user has one bitcoin wallet. All transactions are performed by real bitcoin transactions

Pros: + Simplifies client logic. We do not need to "count" money - just retrieve them from bitcoin wallet and display to user + Way more secure - user have no real access to other uses wallet

Cons: + harder to implement + When we make transaction from one user to another we need to receive confirmation from bitcoin network which is not immediate. + risky. If hacker was able to compromise not account, but access to system same risks as previous options apply

Separate bitcoin wallet + cash-in-out wallet

The main problem of the previous approach is that a possible hacker has access to all bitcoins in the system, including bitcoins used by the cash-in and cash-out wallet. The solution is to run the system using 2 separate modules (ideally on two different network machines) - one for in-system transactions and one for cash-in/cash-out. The second one should accept just two operations "Give X BTC to user Y in system" or "Take X BTC from user Y". Since the code of second module will be simple and straightforward - it's easier to make it secure. Compromising client accounts will not compromise real money

How that will work?

AAA (Authorization, Authentication, Accounting) is always a compromise between usability and security. We suggest the following approach:

User registers with his phone number. He may (or may not) enter recovery email address
When user signs-in into the application two-factor auth is used by default - login, password + OTP (one-time-password) sent by SMS. Two-factor auth may be disabled in the settings by the user
Each transaction higher than X$ is confirmed by OTP. If total sum of transactions within one hour exceeds X$ - each transaction is confirmed by OTP (that is done to prevent stealing money by millions of small transactions)
Password recovery - user will have two options - send recovery info to email or to phone. If he selects phone - recovery information is sent within next 48 hours (but not earlier than 24 hours). That is done to ensure that if phone is stolen - user has enough time to block sim-card.

What is the status of development?

We're currently working on server-side stuff - bitcoin's transferring between two separates system. This part of system will be covered by unit tests to ensure security and reliability. Code should be ready soon if anyone is interested

Bruno Araujo: Tricky Python bugs: comparison madness

2014-03-18T17:21:24+00:00

Python is a great language for a lot of tasks which are not performance sensitive, and even with this kind of task it can somewhat be efficient (think Cython or NumPy). In this post I’ll talk about a problem that can be easily overlooked and can turn into a debugging madness if you’re not aware of it: comparison errors.

For the sake of argument, let’s say you need to implement a function to compare two integers. This can be easily achieved:

def compare(n1, n2):
    return n1 < n2

This function returns True if n1 < n2, and False otherwise. Simple enough, right? There is a crucial detail, though: it will only work properly if both parameters are integers. What about one of them is a string?

In [1]: compare(1, 2)
Out[1]: True

In [2]: compare(2, 1)
Out[2]: False

In [3]: compare('1', 2)
Out[3]: False

In [4]: compare(1, '2')
Out[4]: True

In [5]: compare(3, '2')
Out[5]: True

That may seem an absurd use of the function, but think about a larger problem or system: you will be passing a lot of things around inside variables, and Python just won’t complain if the variables are not of the same type, resulting in a silent and hardly visible bug in your code. In fact, anything can be compared, and the function will happily run and return with no error:

In [6]: compare(list(), 3)
Out[6]: False

In [7]: compare(3, dict())
Out[7]: True

In [8]: compare(list(), dict())
Out[8]: False

In [9]: compare(str(), dict())
Out[9]: False

In [10]: compare('str', dict())
Out[10]: False

This seemingly innocent function will work regardless of what you pass to it, and it takes only one unnoticed wrong parameter to skew the result. The best solution is the obvious one, which is passing the proper types, but if you cannot have this guarantee you can detect these kind of errors using the type() builtin function. An example follows (not the most efficient one, but it will suffice):

def compare(n1, n2):
    if type(n1) != int or type(n2) != int:
        # handle error, e.g. throw an exception
        raise RuntimeError("one of the parameters is not an integer")
    return n1 < n2

In [23]: compare(1, 2)
Out[23]: True

In [24]: compare(1, '2')
--------------------------------------------------
RuntimeError
/tmp/ in ()
----> 1 compare(1, '2')

/tmp/ in compare(n1, n2)
      1 def compare(n1, n2):
      2     if type(n1) != int or type(n2) != int:
----> 3         raise RuntimeError("one of the params is not an integer")
      4     return n1 < n2

RuntimeError: one of the params is not an integer

Always remember to use comparisons with the proper care in Python, or you may end losing a lot of time tracking a nearly invisible problem (true story).

UPDATE: msi provides this helpful bit in the comments:

BTW: This is fixed in Python 3.X:

The ordering comparison operators (<, =, >) raise a TypeError exception when the operands don’t have a meaningful natural ordering. Thus, expressions like 1 < '', 0 > None or len <= len are no longer valid, and e.g. None < None raises TypeError instead of returning False.

Anderson Lizardo: Controle de ponto usando dbus-monitor

lizardo — 2014-01-17T14:55:44+00:00

Aqui no trabalho, a folha de ponto é registrada manualmente, usando uma planilha do Excel onde precisamos registrar as horas de entrada/saída. É arcaico, mas tem sido menos problemático do que o ponto eletrônico, que às vezes dava pane, engatava o rolo, acabava papel etc.

Pois bem, até recentemente, eu vinha coletando a hora de entrada/saída a partir dos logs de sistema da minha máquina (visto que quase sempre eu dou resume/suspend no início/fim do expediente). Mas às vezes eu não desligava a máquina e por isso não tinha aquele dia registrado. Tive então a ideia de monitorar as mensagens D-Bus que são enviadas quando faço o bloqueio/desbloqueio da tela. Para quem não sabe, o D-Bus é a ferramenta mais usada no Linux para comunicação entre processos (InterProcess Communication, IPC). Por exemplo, sempre que a tela é bloqueada, certa mensagem é enviada via D-Bus e os processos interessados ficam “ouvindo” por esta mensagem. O mesmo ocorre no desbloqueio.

Sem mais rodeios, este é o script que escrevi:

#!/bin/bash dbus-monitor --profile \ "type=signal,path=/org/gnome/SessionManager/Presence,\ interface=org.gnome.SessionManager.Presence,\ member=StatusChanged" | \ awk '$7 != "NameAcquired"{system("date -d @"$2)}'

O dbus-monitor permite “ouvir” as mensagens D-Bus que foram autorizadas para o usuário. A opção --profile altera o formato da saída de modo que cada mensagem D-Bus seja expressa em uma linha. Exemplo de saída:

sig 1389965062 467981 2 /org/freedesktop/DBus org.freedesktop.DBus NameAcquired

Estamos interessados na segunda coluna, que é um “timestamp” que pode ser convertido para o dia/hora em que a mensagem foi recebida. No script, o comando date é usado para converter esse timestamp para uma data legível. Por exemplo, date -d @1389965062 gera Fri Jan 17 09:24:22 AMT 2014 no meu fuso horário (UTC-4).

O awk é uma poderosa ferramenta para filtrar e processar texto, especialmente se este tiver um formato tabular como o gerado pelo “dbus-monitor –profile”. No script, ele simplesmente lê a segunda coluna de cada linha e executa o comando “date” (como descrito acima) para converter o timestamp para uma data legível. Notem que ele exclui a linha onde a sétima coluna seja igual a “NameAcquired”. Essa linha sempre é gerada quando se inicia o dbus-monitor e não representa o evento que nos interessa. Para entender melhor o funcionamento do awk, sugiro ler o bom e velho manual.

Só resta explicar o significado do texto:

"type=signal,path=/org/gnome/SessionManager/Presence,\ interface=org.gnome.SessionManager.Presence,\ member=StatusChanged"

De acordo com o manual do dbus-monitor, esta é uma “watch expression”. Basicamente, é um filtro que descreve que tipo de mensagens queremos observar (as demais são ignoradas). Em português, a linha acima diz que estamos interessados em sinais (type=signal) cujo caminho seja /org/gnome/SessionManager/Presence, interface org.gnome.SessionManager.Presence e nome StatusChanged. Mais detalhes sobre o significado dos temos usados aqui podem ser encontrados no tutorial do D-Bus.

Faltou ainda explicar como eu uso o script. Por enquanto, eu inicio ele manualmente usando este comando (que pode ser digitado em qualquer terminal):

./scripts/track_login_logout.sh &>> ~/ponto.txt &

Usei algumas funcionalidades do shell que talvez não sejam do conhecimento geral, então vale explicar. “&>> ~/ponto.txt” siginifica “redirecione tanto a saída padrão como a saída de erro para o arquivo ~/ponto.txt. O “&” ao final da linha significa que o comando irá rodar em background, portanto a janela do terminal poderá ser fechada e o script continuará em execução.

Uma última observação: a mesma técnica pode ser usada para monitorar outras mensagens D-Bus, basta trocar a “watch expression”. A que eu usei depende do GNOME/Unity, e foi testado apenas no Ubuntu 12.04.

Morpheuz: Open Academy and KDE

2013-12-26T11:45:45+00:00

It has been a while that I don’t write here but I thought this subject was worth it

Open source is a huge part of Facebook engineering. Whether we’re building new data infrastructure tools like Corona and Presto to manage our warehouses on Hadoop, releasing a new mobile build tool like Buck, or improving PHP runtime with HHVM, open source projects are integral to our operations.

The hypothesis Facebook is exploring with Open Academy (I love the fact that we have Akademy already :P) is that the best way to learn about software engineering is to do software engineering. As simple as that sounds, we have not traditionally done a good job with this in academia. CS departments may provide a project experience as part of the curriculum, but it typically does not map well to a real world software engineering experience. Facebook and the partner universities wants to do better.

To help bridge this gap between school and industry, last spring Facebook teamed up with Jay Borenstein, a computer science professor at Stanford, to launch Open Academy. Open Academy is a program designed to provide a practical, applied software engineering experience as part of a university student’s CS education. The program works closely with key faculty members at top CS universities to launch a course that matches students with active open source projects and mentors and allows them to receive academic credit for their contributions to the open source code base.

The idea is that partnering with selected open source projects holds a great deal of promise on a number of levels; it will give students exposure to learning how to come up to speed in an established code base, revision control, project estimation and access to examples of good software engineering practices beyond what we are able to provide inside university walls, among many other things.

One question that might be on top of your mind is: “how is this different than Google Summer of Code?” One key difference is the tight integration with universities. Facebook OpenAcademy runs as a university course offering for which a student receives a grade and academic credit. It also allows the university teaching staff to stay very involved and work in tandem with the open source mentors to give students good support for their software development efforts. Another difference is the team element. This course offering partners students from around the globe on teams that work together, which IMHO is exactly the way KDE works.

In a nutshell, the program works like this:

A student team of 3-8 students, potentially spanning multiple universities, is formed
A matching process is run that puts a student team with an open source project
The team and a “mentor” from the open source project are flown to a location for a weekend ramp up hackathon (the location is Facebook headquarters in Menlo Park, CA). The students can work on improving the project in two ways: by knocking out low hanging fruit issues that already exist or by identifying and pursuing new functionality
The students will work on projects for between 8-20 weeks depending upon the schedules of the universities involved Univerisity faculty will be closely involved and consult with the mentor when determining final student grades. Expectations will be set with the students that states the coding contributions they make – the quality and ambition of them – will be the main factor determining their grade
Start date: January-Feb depending on university
Feb 7-9 Hackathon at Facebook Headquarters in Menlo Park, CA
Midterm Acknowledgment (tbd)
End date: Mar – Jun depending on university

Summary of expectations for mentors:

Attend Kickoff Hackathon at Facebook HQ Feb 7-9, 2014 (your flight and hotel will be paid for)
Meet the students working on your project(s)
Help the students select project goals and develop plans to achieve them
Set expectations for how you like to communicate with the team
Be responsive to requests from the team during the course
As often as possible, attend weekly, 30 minute remote team meetings (IRC, Hangout or equivalent)
Be prepared to provide the course instructor with a grade recommendation at the end of the course.
Overall time comittment: You are obviously in control of this and it is also proportional to the number of students you decide to have, but not counting the hackathon, 3-5 hours per week is our expectation

I am asking every interested mentor to fill in project proposals here: http://community.kde.org/Open_Academy . I am excited that KDE can have some slots in this program and as with every other program out there that brings students to open source I am sure KDE is the best place for a student to learn more about software engineering.

If you have questions, feel free to email me.

Everton Arruda: Get a list of TODOs and FIXMEs in your project

2013-10-21T12:51:37+00:00

If you are like me and add FIXMEs and TODOs in your codes, this is a good tip.

To get the list of TODOs and FIXMEs, run the following command:

grep --exclude-dir=.git -rEI "TODO|FIXME" . 2>/dev/null

This command makes a recursive search in all directories and files inside you current directory for FIXME or TODO patterns.

You can make a one word command with that line, just add the following to your ~/.bashrc and reload it:

alias tasks='grep --exclude-dir=.git -rEI "TODO|FIXME" . 2>/dev/null'

The word I chose was ‘tasks’, so, whenever I run ‘tasks’, the command is executed.

Source: http://www.commandlinefu.com/commands/view/12833/get-a-list-of-all-todofixme-tasks-left-to-be-done-in-your-project

Danilo Cesar: YouCompleteMe: semantic auto-completion and JumpToDefinition for VIM on webkitgtk+

2013-09-11T20:27:25+00:00

Prelude

Jump to the next session if you don’t want to read lullabies

There was a time when I was a Qt application developer working with vim. It worked pretty well until I started to work with large applications such as Meego Application Framework.
Hundreds of thousands of files, classes, I couldn’t memorize everything. Then I discovered QtCreator with VI-mode.

I’d never really liked big/heavy IDEs but QtCreator wasn’t that heavy, it had a decent vi-mode and I was able to use the cool things such as auto-completion and jump-to-definition.

Three years later here I’m working with raw VIM again, on top of webkitgtk+ so QtCreator is not an option anymore. What to do?

You can tell me about ctags and cscope… Sorry, ctags and cscope sux for c++ code, really! I tried hard, I wrote scripts to parse the GNUMakefile.list.am and use only the important files but it wasn’t enough. Ctags can’t really deal with classes, doesn’t know much about scopes and VIM auto-completion wasn’t semantic. How to solve that problem all-in-once?

YouCompleteMe

Description

From the original website: “YouCompleteMe is a fast, as-you-type, fuzzy-search code completion engine for Vim.”

Installation

The installation procedures are quite simple and well defined on https://github.com/Valloric/YouCompleteMe#readme
Basically, you need to use Vundle (why are you not using it yet?!) and drop the YCM files in the right place. The installer will do the rest.

vimrc and vundle

if YouCompleteMe is your first Vundle plugin, your .vimrc file should start like this:


set nocompatible               " be iMproved
filetype off                   " required!

set rtp+=~/.vim/bundle/vundle/
call vundle#rc()

Bundle 'gmarik/vundle'
Bundle 'Valloric/YouCompleteMe'

Remember to use those lines at the beginning of the file, other rules should appear after that.

How it works

Basically YouCompleteMe uses a .ycm_extra_conf.py, which is a python script that helps YouCompleteMe plugin to find the compiling flags. Those flags are required so clang can interpret #includes and #defines from other files. It might be required a .ycm_extra_conf.py for each project you work on (Although it would be awesome to have a generic Makefile parser).

For webkitGtk, you can drop this file into the root folder and you’re done. Or you can help me to add traction to Ycm and push this patch upstream.

Tips&Tricks

Add this to your .vimrc


nnoremap <leader>y :YcmForceCompileAndDiagnostics<cr>                                                            
nnoremap <leader>pg :YcmCompleter GoToDefinitionElseDeclaration<CR>
nnoremap <leader>pd :YcmCompleter GoToDefinition<CR>
nnoremap <leader>pc :YcmCompleter GoToDeclaration<CR>

If you’re used to the webkit codebase you know how painful it is when you touch one file, wait two minutes to build and find out that you forgot a semicolon.
You can type \y to check if the file is compilable (clang will actually compile the file for you) and give you warnings/errors for that specific file.
\pc and \pd are also usefull for jump to definition/declaration.

So, if you’re working on WebkitGtk now (or any other project that might benefit from YouCompleteMe), give it a try!

Bruno Araujo: Unity: from zero to Pong

2013-08-22T02:47:22+00:00

In my last post I’ve talked about some game APIs, and could achieve some progress with MonoGame: an animated sprite, controllable with an X360 controller. It took some effort but I’ve learned a lot of things and also regained some knowledge of C# (I’ve worked as a .Net developer a long time ago). But from there, I realized that writing a game practically from scratch is no walk in the park, and even if I were able to spend my whole day coding I’d still need a immense amount of time and effort to do something even remotely playable.

So, why not stand in the shoulder of giants? Enter Unity, which in their words is a “game development ecosystem”. Although it is meant for 3D games, it is still possible with a bit of effort to make 2D ou 2.5D ones, and it uses C# (also JS and Boo). A perfect fit for my next step of doing, er, something.

After searching for some tutorials and watching these videos I was able to make a basic Pong clone in the space of three days. I’ve put the project online at GitHub (https://github.com/baraujo/pong-clone), and this is what it looks like:

You can try for yourself on http://baraujo.net/pong. I was able to run it on WIndows, Linux, Android and the Web player, where I took the screenshot above. Running it on Android was tricky because of the touchscreen, which makes the paddle control not so trivial, but with a bit of math and patience it became somewhat playable. I intend to keep improving it, adding a big score display, just like the classic Pong, an improved CPU player (and the option to have a second player in the same keyboard/mobile) and various adjustements. I’ll update this post when these improvements are finished.

Bruno Araujo: short story: a single shot

2013-07-13T20:40:08+00:00

So, while I’m still gathering material for the technical post, I’ll leave here a short story written in a very idle moment. It’s written in Portuguese (my native language, BTW), maybe one day I’ll translate it to English, but not today.

*.*.*

a single shot

Essa era a sua chance. Sua última chance. Caso acertasse, poria fim a esta situação inteira, salvaria diversas vidas, poderia ganhar diversas condecorações e ainda sairia como o heroi da história, pela primeira vez em toda a sua vida.

Exceto que não seria apenas a primeira, mas também a última, e ele não seria um heroi, mas um mártir.

Esta guerra está perdida. Escondido nas sombras, com o que restou dos seus companheiros, os pensamentos aceleram e uma ideia começa a ganhar força. Ele poderia incapacitar permanentemente o acampamento inimigo, mas a qual custo? Uma missão suicida era a única alternativa, se esgueirar durante a noite e plantar explosivos no depósito de armas. A confusão seria tamanha que um ataque surpresa destruiria o acampamento de homens cansados e mal nutridos, não muito diferentes dele mesmo, e garantiria a passagem pela fronteira, a apenas alguns quilômetros naquela direção, objetivo que poderia ser cumprido antes do amanhecer. Quanto a ele próprio, estaria no meio de um monte de soldados apenas esperando para puxar o gatilho na direção de alguém com um uniforme ligeiramente diferente. E não demoraria até que isso acontecesse, teria sorte se tivesse uma morte rápida.

Mesmo com uma grande chance de sucesso, seu plano enfrentara a resistência ferrenha dos seus compatriotas. “Não vale a pena se arriscar por algo que tem pouquíssimas chances de dar certo”, dizia um deles. De fato, caso o plano não funcionasse, ele já tinha pleno conhecimento que seus inimigos não costumam fazer prisioneiros de guerra.

O que fazer? Ele teria que agir duplamente escondido – dos inimigos e de seus compatriotas. Só poderia contar consigo mesmo para planejar e executar o próprio plano à revelia de todos. Precisava pensar rápido porque o dia se aproximava e então já seria tarde demais.

…

Quase todos estão no chão, alguns dormindo e outros exaustos. Poucos estão de pé, vigiando. E ele, passando ao largo de todos sem alertar uma única alma da sua presença. Nunca o treinamento de guerra na selva tinha sido tão útil, ele era praticamente uma sombra passando de uma árvore a outra. Carregava a mochila secretamente preparada, com explosivos suficientes para derrubar um pequeno prédio, e ia em direção ao seu objetivo. No acampamento inimigo uma situação semelhante ao seu próprio acampamento, com poucos de pé, e ele facilmente chega até a larga tenda abrigando vários tipos de munição e granadas. Joga a mochila lá dentro, vai até a distância máxima que o transmissor funciona, algumas dezenas de metros, se põe atrás de uma árvore, e fecha os olhos. Em seguida, fecha o mecanismo que carregava na mão direita.

Era claro como o próprio sol, e ruidoso como um vulcão. Os companheiros levantam-se repentinamente sem saber o que fazer naquela confusão mas em poucos segundos tudo fica claro. O plano suicida tinha sido executado. Nada mais poderia ser feito em relação ao futuro mártir, senão reunir o que restava de forças e partir com tudo pra cima do acampamento completamente desordenado. Em questão de minutos, os poucos soldados agindo de vigias foram facilmente alvejados e os restantes rendidos facilmente ou derrubados como moscas. Enfim, uma vitória, uma pequena vitória no meio daquela guerra. E praticamente todos conseguiram ultrapassar as trincheiras e cruzar a fronteira pesadamente defendida pelos seus aliados.

Praticamente. Só um soldado não cruzou a fronteira, e foi dado como desaparecido. Ao invés de condecorações, uma condenação à revelia por insubordinação grave e deserção, crimes que o conduziriam à corte marcial caso estivesse presente. Fuzilamento. Ao invés de herói ou mártir, insubordinado e desertor. Ele havia se tornado um pária, rejeitado no seu próprio país e caçado como um animal pelo país inimigo.

O clarão da explosão mostrou a ele um pouco da selva além, onde ele podia observar um riacho e um pequeno bote amarrado às margens. Desamarrou o bote e seguiu em frente, ciente de todos os crimes que havia cometido aos olhos dos seus compatriotas. Sem ter pra onde ir, passou dias na mata, utilizando tudo o que aprendeu no curso de sobrevivência na selva para manter-se vivo. Quando estava já à beira do colapso físico e mental, chegou a uma pequena vila, isolada e alheia aos acontecimentos dos dias anteriores, onde foi recebido simplesmente como um homem perdido na mata e nada mais. Foi dado a ele um lugar para dormir e algo para comer, e depois de muitos anos ele foi tratado não como subordinado ou inimigo, mas como igual. E decidiu, naquele momento, iniciar uma nova vida, naquele lugar distante onde o horror da guerra não havia ainda chegado.

…

Vários anos depois ele era apenas um nome num arquivo morto do tribunal militar do seu país, e o inimigo havia desistido de procurá-lo. Mas sabe-se que ele viveu um por um bom tempo depois disso e que talvez ele até tenha formado uma família, segundo relatos dos habitantes da antiga vila – hoje cidade – onde ele passou o resto dos seus dias. E finalmente ele foi reconhecido como um herói de guerra devido a quantidade de vidas que ajudou a salvar. Apenas com o tempo ele teve o seu esforço reconhecido, assim como muitos.

…

No breve momento em que sua mão fechava o mecanismo, só havia um pensamento na sua cabeça: “Essa é a minha única chance de salvar, ao invés de destruir”.

Bruno Araujo: Developing games: the first steps

2013-06-23T08:11:41+00:00

So, after playing great indie games like Fez, Super Meat Boy and Braid, and also seeing a documentary about the people behind them, I finally decided to start making games, instead of just playing them. In fact, the real motivation for me to become a Computer Scientist was just that: make games. But as time passes, so does your goals, and sometimes you’re too distracted by the other aspects of your life to remember the ideals of the past. And so, I studied many years to become a Scientist and worked other many years accumulating experience as a software developer, working on projects big and small, open source and proprietary ones.

I’ve played hundreds, maybe a thousand games over the course of my life and various videogames and PCs, and around 9 years old I was copying BASIC games from small books into my Hotbit computer, watching in awe how these hours of inserting code could become something I could control and interact. Fast forward 21 years and I’m here again, with modern computers, languages and algorithms, but the same purpose: transform code into something I can play. But with a fundamental difference: I’ll make my own games now.

After searching for some tools, these are the ones I’m trying to learn and code right now:

cocos2d-x: A multiplatform library for making 2D games. As a port from cocos2d-iphone, its focus seems to be mobile platforms, and most of the examples and tutorials around the ‘Web talk about just that. Its C++ API looks very complete but also very complex, with classes to do almost anything for you; lots of examples show the power of this framework for the ones willing to learn.
Polycode: According to the official Web page, “a free open-source framework for creating crossplatform games & interactive applications”. It has C++ and Lua APIs, based on various open source libraries, along with an IDE for managing game content and developing Lua programs. Although it is not quite yet finished its API looks very nice, with less verbosity than cocos2d-x for the same tasks; while I wait for a more proper release, I’m learning to do some simple examples. Has 2D and 3D capabilities.
MonoGame: A open source implementation of Microsoft XNA, which itself is a API for game development in C#. Most of my life I’ve worked with Linux and open source software and platforms, and although technically I could use MonoGame in Linux, it is more easily done using VS2010/2012 on Windows, which in itself is a departure from these many years. But, as I started to learn how it works, I’m trying to be more open-minded and not dismiss it right away because it is a port of a Microsoft technology; instead, I’m approaching MonoGame with a positive attitude and trying to do something concrete with it. As a matter of perspective, Fez and Bastion were made with XNA and later ported to other platforms with MonoGame.

I’m approaching each one of them in turn, trying to make a simple movable sprite in each API. So far, I’ve managed to do it with MonoGame, which is where I could advance more, and stumbled in some hurdles in cocos2d-x and Polycode. Still, I think these are all powerful tools for easing the process of making a game, and their creators have invested (or are still investing) a lot of time and work. Maybe I could use other tools in the near future or join skilled people with the same goal, allowing me to focus on the programming itself. But for now, back to the drawing board and to the code editor.

Bruno Araujo: Parsing a very large XML file with Python

2013-05-08T05:29:08+00:00

In the process of doing some experiments, I was looking for a suitable text corpus to use; the ideal candidate would have a few GB and would not cause any monetary loss on my wallet. Stumbling around I found out that Wikipedia has various dumps of its pages, and decided that the 2013 dump would be a suitable bag of text for my experiments – a single XML of 9GB, which by the way is the compressed size. The actual size is 42GB. I had in my computer the largest XML file I’ve seen in my life, and I didn’t have space left in disk to even extract it – parsing it would be a hell of a job.

Fortunately I already knew how to parse XML with the nice lxml Python module, but how to proceed when I just cannot extract the file? It turns out that there is a module called bzr which exposes an interface for opening .bz2 files in the same fashion the open() function does. After some hours of struggling, searching around several sites, blogs and forums and filling all of memory + swap more than once, I managed to parse the file without blowing up the RAM with this Python script:

from lxml import etree
import sys
import bz2
import unicodedata

TAG = '{http://www.mediawiki.org/xml/export-0.8/}text'

def fast_iter(context, func, *args, **kwargs):
    # http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
    # Author: Liza Daly
    # modified to call func() only in the event and elem needed
    for event, elem in context:
        if event == 'end' and elem.tag == TAG:
            func(elem, *args, **kwargs)
        elem.clear()
        while elem.getprevious() is not None:
            del elem.getparent()[0]
    del context

def process_element(elem, fout):
        global counter
        normalized = unicodedata.normalize('NFKD', \
                unicode(elem.text)).encode('ASCII','ignore').lower()
        print >>fout, normalized.replace('\n', ' ')
        if counter % 10000 == 0: print "Doc " + str(counter)
        counter += 1

def main():
    fin = bz2.BZ2File(sys.argv[1], 'r')
    fout = open('2013_wikipedia_en_pages_articles.txt', 'w')
    context = etree.iterparse(fin)
    global counter
    counter = 0
    fast_iter(context, process_element, fout)

if __name__ == "__main__":
    main()

Update (06/07/2013): This code is now available as a Github gist.

The output format is a single file with one document per line. Probably it would be better to put all these lines in a DBMS or a Berkeley DB file, as the lines can be really, really big, but for now the plain text format will suffice. Every document is normalized to contain only ASCII characters, avoiding potential encoding problems, but besides that no further processing is done. I parsed only the tag “text” from the XML, but if you need to parse more than one tag change TAG into a dict:

TAG = {
    '{http://www.mediawiki.org/xml/export-0.8/}text',
    '{http://www.mediawiki.org/xml/export-0.8/}title'
}
(...)
        if event == 'end' and elem.tag in TAG:

The URL enclosed in brackets is the namespace of the tags, which lxml annoyingly insists in using, so I just ended using the “full name” of the tag I needed. Also, I know that globals are evil, I just wanted to easily count the number of documents processed and I was too lazy to properly write a class. I killed the process when the output file reached around 20GB, which I considered big enough for my experimental needs. Now all that is left is to actually run the benchmarks and see what happens.

And just for the record: I’m not a Python specialist, feel free to point eventual problems or bottlenecks on this code.

Adenilson Cavalcanti: Considerations on home office

savago — 2013-04-24T05:27:09+00:00

Ah… home office. The dream of any real programmer.

I was about to say ‘hacker’ but it seems that this substantive has become something that can’t be said in some circles in current ‘social sensitive’ society (but that is a topic of another post).

Can you imagine that: wake up at any time, no need to commute, have your very own office space decorated in any way you want?

I have being working in a home office setup for the last 16 months and realized a couple things that may be interesting to share with my friends and the opensource community overall.

First lesson: it requires discipline. There is no one around to push you to do your job, so unless you are a disciplined lad, probably you may get in trouble. I’ve lead teams of developers in the past, and I can say that not everyone is cut to work in a more relaxed and self managing environment. What is interesting: the majority of people when offered the opportunity to work remotely will be scared of it and will find an excuse to avoid it.

Second lesson: timezone differences are your friend. Last year I was able for 2 months to wake up early in the morning, visit a beach from 6AM to 12PM and then start working. If you have the opportunity of working remotely for a sponsor located in a different timezone, it is beneficial for both sides to explore that. You will be available at the sponsor’s timezone and at same time, it gives you freedom to do things that won’t be possible in a common 9AM to 5PM workday. Tip: since I’m using KDE, I have in my workspace analog clocks displaying the timezones of all involved parties in a project.

Third lesson: have your office. To properly turn on your mind set, have an office (which may be a room in your house) that will be your work place. When you start working, close the door and make clear to everyone that you are busy and not available. And when you are done with your daily journey, simply leave the place and close the door behind you, to help you change the mindset and be able to relax.

Fourth lesson: communication rules. Be in touch with your pals, provide daily updates and try as much is possible to be aware to what is happening in your project. Being thousands of Kms (or should I say miles to the Imperial guys?) away makes your work pretty hard to be noticed by managers and at same time provides unique challenges to be able to feel how a project is going and what the sponsors priorities really are.

Fifth lesson: you will always loose something. The trash talk at the coffee place? Not documented good practices? Some cool pet project developed by a co-worker? You will loose all that working remotely and that is a sad fact that you need to accept. There are also limits to what you can contribute back to your company while working remotely, which leads to the next point.

Sixth lesson: you will achieve the ladder’s end pretty fast. We, humans, are related to primates. Hunters in the past used to depend on facial expressions to properly assess danger and happiness. Even with current’s advances on remote communication, there are limits imposed by our own nature that won’t be overcome by technology. You need to accept that you won’t be able to achieve the top of your company while working remotely.

Seventh lesson: it requires courage and confidence on your skills. Think about it: you probably signed a contract with a foreign company that probably cannot be enforced in your local country and which gives freedom to both parties to end the aforementioned contract at any time. I personally feel that this keeps things straight and honest between both parties (you need to be happy and the sponsor needs to be satisfied with the results), but there are people who may be afraid of such situation.

Well… this is it. If I remember other things I’ve learned on this subject, I will update this post. Do you have thoughts on the subject? Let me know.

Ademar de Souza Reis Jr.: Retrospectiva 2012, planos pra 2013

2013-01-19T04:14:28+00:00

Outro dia me perguntaram se o blog estava morto, pro que respondi: “só estará morto quando eu parar de postar a retrospectiva de fim de ano”. Bem, esse ano eu passei perto: o tradicional post de fim de ano está mais de duas semanas atrasado. Mas antes tarde do que nunca, aqui está ele! :-)

No ano passado eu disse que 2011 havia sido um ano “excelente, pra ser lembrado”. 2012 não foi nada diferente: a Sofia chegou e fizemos mais uma grande mudança, saindo de Recife – PE e vindo pra Canela – RS.

Sofia – O destaque do ano
A Sofia nasceu em Maio e está crescendo saudável, linda e cada vez mais divertida. Eu a cada dia fico mais apaixonado pela filhota. É ótimo acompanhar as mudanças quase que semanais e a evolução de cada etapa. Um grande aprendizado pra eu que tinha pouquíssima experiência com bebês e um grande desafio pra eu e a Viviane como pais. Ter um filho tem sido uma experiência incrível, fortemente recomendada a todos (ela fechou o ano com 7 meses e 20 dias de idade).

Sofia aos 5 meses, passando frio na Serra Gaúcha

Sofia em seu sétimo mensário.

Trabalho – Red Hat
Falando em aprendizado, o ano foi puxado, mas muito produtivo no trabalho. Minha equipe é de um nível altíssimo e liderá-los tem sido um desafio enorme. Em 2012 eu aprendi muito e conheci pessoalmente vários colegas em ótimas viagens pra Israel, EUA (Westford, MA), Espanha (Barcelona – KVM Forum) e São Paulo.

Mosaico com algumas fotos da viagem à Israel

Mudança
Quem me conhece há vários anos sabe que eu levo uma vida de nômade: desde o casamento em 2006, já nos mudamos 4 vezes (Curitiba, Manaus, Curitiba de novo e Recife). Depois de 4 anos em Recife, era hora de mudar novamente. Dessa vez a escolha foi criteriosa: trabalhando de casa eu tinha a opção de escolher qualquer região/cidade no Brasil pra morar. A escolha foi pela Serra Gaúcha e a cidade originalmente escolhida era Caxias do Sul. Acontece que após uma visita que era pra ser breve, acabamos optando pela região de Gramado/Canela e resolvemos vir pra esta última (sob forte influência do Fabiano e da Pricila, amigos que fizeram essa migração alguns anos antes).

A escolha da moradia é que não foi trivial: resolvemos tomar coragem e comprar nossa própria casa (chega de apartamento) e após muita procura, encontramos uma do nosso gosto, mas ainda em acabamento. A vantagem foi poder escolher os detalhes, trocar materiais de acabamento, erguer e derrubar paredes (Viviane é Designer de Interiores). A desvantagem: ter que esperar. Após consideráveis atrasos, a entrega está planejada para Fevereiro. No período entre a saída de Recife e a escolha da casa, passamos algumas semanas morando na casa de amigos e parentes (valeu Saulo/Pati, Fabiano/Pricila e Lico/Lisi), até finalmente alugarmos um pequeno apartamento pra receber a mudança e nos estabelecermos temporariamente.

Estudos – Mestrado
Nessa eu “arriei”: FAIL. Não dei conta e tive que trancar o curso. A combinação Sofia + Trabalho e a ideia da mudança me tirou a possibilidade de continuar os estudos na UFPE. A essa altura do campeonato, não sei se vou retomar o mestrado algum dia. Quem sabe um MBA num futuro próximo.

Viagens
Além das viagens à trabalho, não fiz nenhuma viagem de férias significativa, pra algum local desconhecido ou novo. Mas morando aqui no sul foi mais fácil visitar a família: passamos o Natal em Rio Grande – RS, onde toda a família da parte da Viviane estava reunida e a virada do ano em Umuarama – PR, com a minha. Na volta desta última, fomos até Foz do Iguaçu pra fazer umas compras no Paraguay. Foi a primeira viagem internacional da Sofia. :-)

A lista de objetivos de 2012 foi modesta e genérica, se resumindo a “ser um bom pai e marido”, “ser um bom profissional” e “ser um bom estudante”. Como já mencionado, falhei neste último item, mas acho que consegui meu objetivo nos outros dois. Uma coisa que estava na lista de 2011 finalmente saiu: atualizei os álbuns de fotos do site com breves relatos de viagens: Patagônia: principais fotos; Fotos da Patagônia: Região dos 7 Lagos (Bariloche); Fotos da Patagônia: Parque Nacional los Glaciares (Geleira Perito Moreno); Fotos da Viagem pela Europa; Fotos de Oslo – Noruega e Fotos de Nova York.

Ah, e antes de sairmos de Recife, fechamos algumas outras pendências: fomos até o Instituto Ricardo Brennand e ainda levamos a Sofia (com 4 meses) pra um primeiro mergulho na praia de Porto de Galinhas, referência do estado onde ela nasceu. :-)

Pra 2013 a lista é mais detalhada:

Explorar a região de Canela e Gramado: ainda não visitamos nem os principais pontos turísticos
Entrar em forma — bike, caminhada, academia… qualquer coisa
Agora que moro na serra, quero voltar a acampar, subir montanhas e etc:

Acampar no mínimo 3 noites, sendo uma delas com a Sofia
Descer o Rio Paranhana de Rafting
Fazer pelo menos 2 trilhas de Mountain Bike

Estar com a casa “pronta” (incluindo obras ainda em planejamento: área de serviço, ante-sala e garagem)
Visitar um país que eu ainda não conheça
Montar um álbum com fotos da região

Então é isso, missão cumprida: o blog não está morto. :-) Feliz 2013 a todos!

Link Permanente | 4 commentários
Post tags: retrospectiva

Bruno Araujo: Welcome! Welcome to City 17…

2013-01-05T06:20:37+00:00

Yeah, I know I’ve used this little Half-Life joke before, but it never gets old. Anyway, this will be here for some time, while everything is being calibrated. Ciao!

Adenilson Cavalcanti: icecc setup how to

savago — 2012-11-23T14:58:32+00:00

I wrote this article back last February just to document the way I did setup icecc in my home and decided to put it here for easy access. Just ignore the time of compilation for both Qt and webkit (since then, both have changed considerably and I also added another node into the cluster, an Intel i7). For reference, the trick of combining ccache and icecc is not very well know and this is why I think it is still worthwhile to document it here.

Maybe those are techniques that you are already familiar with, so my advanced apologies. If you don’t have used icecc/ccache before or just want to known a little more about it, let’s proceed.

Ok, so this is the scenario: you are working with a big code base or maybe you need to recompile a specific version of Qt/WebKit. Or it can be another reason, maybe you just want to self compile your own kernel/desktop like real programmers do, right?

Even with today’s multicore/gigabyte based systems, it may take a little longer than you wish to compile it all: http://xkcd.com/303/

So, what to do in this cases? There are some projects that can help you out, let’s examine the case of ccache first.

Ccache (http://ccache.samba.org/) is a tool created by samba team, that project that offers CIFS/SMB protocol implementation over *nix kernels. It works with a simple principle of generating a cache of compiled files and reusing those object files if no changes in the source code were detected. Instalation is quite easy in Linux (sudo apt-get install ccache), and you only need to point your PATH environment variable to ccache.

An alternative is to have a file in your home (e.g. ~/.samba.bashrc) with the following function (and source it in .bashrc):
function sambacache {
export PATH=/usr/lib/ccache:$PATH
}

So that you can enable samba by doing:
adenilson@macmini:~$ which gcc
/usr/bin/gcc
adenilson@macmini:~$ sambacache
adenilson@macmini:~$ which gcc
/usr/lib/ccache/gcc

The first compilation will take the same time as you are used to, but the following ones (after the cache is populated i.e. check ~/.ccache directory) will give huge speed ups. So, as an example, a favorite of mine (http://cellardoor.googlecode.com) takes 4s to compile in a Intel i5@2.3Ghz the fist time but next it will take only 0.48s to recompile after a make clean. Pretty neat, hum?

Ccache adds almost no overhead in the compilation and is a perfect helper if you have just 1 computer to do your compilation jobs.

There is an issue with the way that ccache works. It is easy to see that if your project has many self-generated files by each new compilation, the cache will not be reused. So, if for recompiling just Qt there are not big benefits (it self generates lots of header files at beginning of compilation), for WebKit it brings the time compilation down to just 2 minutes.

The next project is icecc (http://en.opensuse.org/Icecream) and it allows to use several computers in the same network to distribute the compilation jobs. It is the ‘evolution’ of distcc, offers some extra features and has a great monitoring tool called ‘icemon’. Instalation is easy in Linux (sudo apt-get install icecc icecc-monitor), configuration not that easy and the documentation could be better.

The speed ups are almost linear, being the network speed the most probable bottleneck. I’ve used it in my previous job with up to 15 computers connected and it allowed me to do a cold recompile of Qt in less than 8 minutes (bear in mind that those 15 computers were also compiling other projects at same time…).

To make it easier to follow, I will explain my home setup. I have 2 machines:
a) macmini: Intel i5@2.3Ghz running Linux 32 bits (dual boot with OSX Lion).

b) blackbloat: Acer notebook (cheapest one I could find, so portability is not its strongest feature…) Intel i5@2.67Ghz running linux 64 bits (dual boot with Windows 7). IP is 192.168.1.135

What is neat about icecc is the fact that you can have varied nodes in your cluster and still perform compilation jobs distribution. It does that by allowing you to create a ‘rootstrap’ of libraries and compilers to be used, being those sent to the other nodes in your first compilation. To do it, you should run:

adenilson@blackbloat:~$ icecc –build-native
adding file /usr/bin/gcc
adding file /lib/x86_64-linux-gnu/libc.so.6
adding file /lib64/ld-linux-x86-64.so.2
adding file /usr/bin/g++
adding file /usr/bin/as
adding file /usr/lib/libopcodes-2.21.53-system.20110810.so
adding file /usr/lib/libbfd-2.21.53-system.20110810.so
adding file /lib/x86_64-linux-gnu/libz.so.1
adding file /lib/x86_64-linux-gnu/libdl.so.2
adding file /usr/bin/cc1=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/cc1
adding file /usr/lib/libmpc.so.2
adding file /usr/lib/libmpfr.so.4
adding file /usr/lib/libgmp.so.10
adding file /usr/bin/cc1plus=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/cc1plus
adding file /usr/lib/gcc/x86_64-linux-gnu/4.6.1/liblto_plugin.so
adding file /etc/ld.so.conf=/tmp/icecc_ld_so_confpM2h21
creating de07a31507267d47693646853b78125e.tar.gz

adenilson@macmini:~$ icecc –build-native
adding file /usr/bin/gcc
adding file /lib/i386-linux-gnu/libc.so.6
adding file /lib/ld-linux.so.2
adding file /usr/bin/g++
adding file /usr/bin/as
adding file /usr/lib/libopcodes-2.21.53-system.20110810.so
adding file /usr/lib/libbfd-2.21.53-system.20110810.so
adding file /lib/i386-linux-gnu/libz.so.1
adding file /lib/i386-linux-gnu/libdl.so.2
adding file /usr/bin/cc1=/usr/lib/gcc/i686-linux-gnu/4.6.1/cc1
adding file /usr/lib/libmpc.so.2
adding file /usr/lib/libmpfr.so.4
adding file /usr/lib/libgmp.so.10
adding file /usr/bin/cc1plus=/usr/lib/gcc/i686-linux-gnu/4.6.1/cc1plus
adding file /usr/lib/gcc/i686-linux-gnu/4.6.1/liblto_plugin.so
adding file /etc/ld.so.conf=/tmp/icecc_ld_so_confWHgW78
creating 73c127da41faa91bfdbae1faedbb2113.tar.gz

And later point the environment variable ICECC_VERSION to where is the rootstrap tarball. In my case, I added a file in my home (i.e. .icecc.bashrc) and renamed the tarball with the following:

function coolice {
export ICECC_VERSION=/home/adenilson/Desktop/ice_rootstrap.tar.gz
export PATH=/usr/lib/icecc/bin/:$PATH
}

Similar to ccache, icecc requires to add it in the PATH before the standard compiler.
adenilson@macmini:~$ which gcc
/usr/bin/gcc
adenilson@macmini:~$ coolice
adenilson@macmini:~$ which gcc
/usr/lib/icecc/bin//gcc

The final step is to choose which machine will run the scheduler, responsible for distributing the compiler jobs between the nodes. In my case, I picked blackbloat, for making the changes permanent, added in /etc/default/icecc
START_ICECC=”true”
START_ICECC_SCHEDULER=”true”

And all the nodes (including blackbloat, since I doubled it as both a scheduler and a compiler node) should know which network to connect to (I decided to call my network ‘tucks’) thus editing the file /etc/icecc/icecc.conf:

ICECC_NETNAME=”tucks”
ICECC_ALLOW_REMOTE=”yes”
ICECC_SCHEDULER_HOST=”192.168.1.135″

There are other parameters you can tweak. For example, by default icecc will assign a compiler job for each CPU/Core/virtual cpu in your system, so it would run at maximum 4 jobs in each node in the case of an intel i5 (2 cores + hyper thread). You can change that by assigning a value in ICECC_MAX_JOBS filed in icecc.conf. Another tweak is the nice level of each running job process, default is 5 (which is good for not disturbing the normal workflow in a computer, but you can get a slightly better performance by changing it to a lesser value).

To monitor the machines in the cluster, just use: icemon -n tucks (see attached image).

It is important to known if *really* the jobs are being distributed. You can check:
a) If in your machine there are several ‘g++’ named processes (using basically no CPU) and a few ‘cc1plus’ named processes using all the available cpu;
b) If the other node(s) has several ‘cc1plus’ processes;
c) If there is considerable I/O going through the network interface (in my case spikes of up to 2.8MB/s)
d) If icemon shows the jobs moving through the nodes.

I once thought that I’ve configured it all fine and then started ‘make -j30′ with all the jobs running in my machine… needless to say, the machine locked.
:-)

Alright, enough talk, let’s see some numbers… To compile Qt 4.8 from the git repository it takes 34 minutes in macmini, running with ‘make -j6′. By using icecc, and running ‘make -j12′ it compiles in 21 minutes (almost half the time). It is important to remember that there are steps that can not be made parallel (e.g. qmake creating the .pro files, moc running, linking and so on).

For webkit plus Qt5, the numbers are even better: 1h 10m X 19minutes (it is almost 400% faster!). Those 50 minutes *by compilation* can add a lot at end of day.

Finally, it is also possible to combine ccache *with* icecc. To do it, just define your path as: export PATH=/usr/lib/ccache:/usr/lib/icecc/bin/:$PATH. In my desktop, I added the following file in my home:

adenilson@macmini:~/apps/webkit/Webkit$ more ~/.bamf.bashrc
function bamf {
export ICECC_VERSION=/home/adenilson/Desktop/ice_rootstrap.tar.gz
export PATH=/usr/lib/ccache:/usr/lib/icecc/bin/:$PATH
}

Attached you can check how icemon looks like while recompiling Qt 4.8 with 2 different views (starview and gantt view).

Ricardo Salveti: Pre-built images for XBMC (Ubuntu 12.04 based) with hw acceleration finally available at Linaro

rsalveti — 2012-07-30T00:07:48+00:00

Back to the Linaro 12.01 release, we were able to publish a XBMC based image with proper hardware acceleration support for Pandaboard. That release was based on the Ubuntu Oneiric (11.10) release, with a 3.1 based kernel, still using the old multimedia architecture provided by TI (e.g old SGX DDK, Syslink, etc).

Took a while (6 months) for TI to be finally able to rebase the hardware decode/rendering solution to the new architecture, providing quite a few improvements and making the architecture a lot simpler as well. A new SGX DDK was deployed with the functional xf86-video-omap open source X11 driver, kernel and userspace with support for RPMsg, dri2video and a few other fancy technologies that now makes the life of developers a lot easier. We’re still stuck with a few proprietary parts, like the SGX driver itself, but luckly that will also improve over the time, as we’re starting to see more and more reverse-engineer based solutions around (even Rob Clark started one called freedreno).

After TI making all the needed components available for Ubuntu Precise (12.04) during the beginning of this month, I was finally able to get them all integrated and working with the Linaro components and LEB. If you grab the latest LEB provided for Pandaboard, you’ll already be able to enjoy most of the updates available, with a better SGX driver, and a working hardware decode with gstreamer without much effort.

Unfortunatelly our TI Landing Team kernel is still missing a few important multimedia-related bits, so until they are properly integrated from TI’s own kernel tree, you’d need to make sure you’re using the kernel available directly from TI (called linux-image-ti-omap4, also available at our Overlay PPA).

To be able to have a fully functional XBMC as well, we also integrated all the new changes to support a native Gstreamer player (improved a lot by Rob Clark as well, adding EGLImage support), which solves most of the a/v sync issues we had with the Oneiric based release. There are probably a few things missing as well, but at least the patch is not that intrusive anymore, and could probably be accepted by upstream later on.

Want to see how things are looking by yourself? Please check at http://snapshots.linaro.org/precise/pre-built/ti-panda-x11-base/ and download the latest pre-built image available. At the time I’m writing this post I used the one available at http://snapshots.linaro.org/precise/pre-built/ti-panda-x11-base/5/ti-panda-x11-base-precise_linarotv-xbmc_20120729-5.html, and all I needed to do was to extract the img.gz file, and copy it with dd (sudo dd bs=4M if=ti-panda-x11-base-precise_linarotv-xbmc_20120729-5.img of=/dev/sdX).

Boot it up, and you should go directly to XBMC. Give it a try, check how well it’s playing the wide range of different video formats. Bugs are also expected and welcome, so please also make sure to open any bug you might find at https://bugs.launchpad.net/linaro-ubuntu and https://bugs.launchpad.net/ubuntu-omap4-extras-multimedia. We’ll keep improving the support over the next cycles, so also make sure to post which image you’re using when doing your tests.

Enjoy!

Adriano Tinoco: QEdje

atdrez — 2012-06-20T12:14:07+00:00

This is my first post, and I will talk about the QEdje project that will be presented next month in Akademy 2008 at Belgium.
The QEdje is a scripting language used to separate code from interface and to speed up the development of applications with beautiful interfaces.
The basic idea of the project is to port the original Edje (used by EFL – Enlightenment Foundation Libraries) to Qt. We, at INdT (Nokia Technology Institute), are planning to release a QEdje version with basic supports probably on the beginning of the next month.
Our plan is to give full Edje support until the end of the year for Qt. To have the same facilities that we had using Evas/Edje in our last projects, like Canola for example (http://openbossa.indt.org.br/canola/).

The great thing about QEdje is that we can combine the best of both worlds. Using Edje to create beautiful interfaces and still using Qt widgets embedded whenever necessary (for example: inputs, webkit browser, …).

Some of the bottlenecks that we are still facing resides on Qt Canvas. Most of the code generates a great overhead on small devices like n800, and to reach a performance near to the Evas performance, some optimizations must be done inside Qt code. But the great news is that INdT and Trolltech are planning to create a code camp for us, probably in September at Norway.

A small show case developed using QEdje:

Adriano Tinoco: Hidra

atdrez — 2012-06-16T07:58:41+00:00

Yesterday I decided to restart an old project called Hidra. Hidra is a very high-level programming language like Python and Ruby that I created some years ago. The project had only personal fun purpose and the code was never released. Hidra is a full object oriented language like Java but with the advantage of being dynamically typed and to have a more compact virtual machine due that fact. The virtual machine is very similar to JVM but with 5x less opcodes. The original compiler and virtual machine code was written using Java, wich reduces the performance a lot. So the idea is to recreate the virtual machine using C++ and for now I’m planning to recreate the compiler as a prototype using Python to have less code and more flexibility to change the grammar.
I spent the last 2 days recreating the virtual machine in C++ and it’s now almost fully functional, but there are a lot to be done in order to be stable. It needs also a garbage collector and debugging support. I will reserve some hours during weekends to work in the project.

The main purpose of the project now is to have a lightweight interpreter that could be used for C++ applications to embed scripts. This could be useful for description languages like Qt Declarative UI and also to create AI scripts for games. Regarding the lightweight/optimized feature, there is a devil in my shoulder telling me to replace the virtual machine to an AST interpreter, in other words start the project from scratch. Maybe I’ll do this, because it’s the simpler way and it would allow more optimizations to be done. But for now I’ll focus on features and syntax changes to consolidate the language.

Below is a Hello World example using the current Hidra syntax (unnecessarily big just to show a little more).

class HelloWorld
{
    var helloText;

    public static function main(args)
    {
        new HelloWorld("Hello World!").say();
    }

    public HelloWorld(text)
    {
        helloText = text;
    }

    public function say()
    {
        System.println(helloText);
    }
}

As you can see the current syntax is similar to JavaScript, but with a more solid object orientation. I intend to maintain the grammar as simple as possible with some small changes to get a cleaner look (maybe more python like).

I’ll send some posts to give feedback of the progress. Stay tuned!

Ricardo Salveti: Great start at Ubuntu Developer Summit (Q or 12.10) for ARM

rsalveti — 2012-05-10T08:13:46+00:00

This week I’m proudly participating at the Ubuntu Developer Summit to help planning and defining what will the Quantal Quetzal (12.10) release be in the next following months.

As usual I’m wearing not only the Linaro hat, but also my Ubuntu and Canonical ones, interested and participating actively at most topics that are related with ARM in general.

And what can I say after the first 3 days at UDS-Q? Well, busy as never before and with great opportunities to help getting Ubuntu to rock even more at ARM, with current devices/platforms and with the exciting new ones that will be coming in the next few months.

Here are a few highlights from the first days:

Monday – May 7th

Introduction and Keynote
- Great start as usual by Mark, showing the great opportunities for both Canonical and Ubuntu, describing the new target and use cases, and also showing how important Cloud is now for Ubuntu. After that we had, finally, the announcement of a real hardware availability from Calxeda, proving that ARM server are indeed real! (which is a quite important accomplishment)
Schedule displays all working with our member’s boards
- This was the first time that all the schedule displays available at UDS were all covered by the ARM boards provided by Linaro. This time we got Pandaboard, Origen and also Snowball constantly showing the schedule through all the day. Low power and powerful devices all around :-)
Plans for a minimum filesystem for embedded devices
- Discussion to cover all the possible embedded related use cases for Ubuntu, and trying to understand the real requirements for a minimum filesystem (rootfs) for those devices. While we didn’t decide to generate the smallest-still-apt/dpkg-compatible rootfs for our users (as ubuntu-core is already covering most of the cases), we’ll provide enough tools and documentation on how to easily generate them. At Linaro side the Ubuntu Nano image should probably reflect such suggestions.
Identify impact of the switch to pure live images for ARM platforms
- Here the focus was basically to review and understand if we would really continue providing pre-installed based images instead of just supporting live based ones. Having the images provided only at the SD cards are very useful to make the bootstrap and install quite easy, but it hurts badly the performance. As we’re now getting ARM boards that are very powerful in many ways, the I/O bound shouldn’t limit what the users would be able to get from them. The decision for Quantal is to drop support for the pre-installed images, and provide live based ones at the SD cards (think like the live-sd image as we have with CD on other archs), where the user would install Ubuntu the same way as done with x86, and using USB/Sata based devices as rootfs by default.
OpenStack Deployment on ARM Server
- The focus of this session was basically to better understand what might be the missing pieces for a proper OpenStack support at ARM. Quite a few open questions still, but the missing pkgs enablement, LXC testing and support and KVM for a few platforms will help making sure the support is at least correctly in place. After initial support, continuous test and validation should happen to make sure the ARM platforms keeps well supported over the time (which will be better stressed and tested once MAAS/Juju is also supported properly at ARM).

Tuesday – May 8th

Detail and begin the arm64/aarch64 port in Ubuntu
- Clearly the most important session of the day for ARM. Great discussion on how to prepare and start the ARMv8 port at Ubuntu and Debian, by starting with cross-build support with multiarch and later support with Fast Models and Qemu. A lot is still to be covered once ARM is able to publish the ARMv8 support for Toolchain and Kernel, and session will be reviewed again at Linaro Connect at the end of this month.
Ubuntu Kernel Delta Review
- Usual review of the patches the Ubuntu Kernel team is maintaining at the Ubuntu Kernel tree. At Linaro this is important as we also enable the Ubuntu specific patch-set at the packages provided by the LEB, for proper kernel and user-space support. Luckily this time it seems the delta is really minimum, which should probably also start to be part of Linux Linaro in the following month.
Integrate Linaro hwpacks for ARM with the Ubuntu image build infrastructure
- Usual discussion about trying to avoid replicated work that is strictly related with each ARM board we support at both Ubuntu and Linaro. Decision is to finally sync with the latest flash-kernel available at Debian and try to get the common project/package with the hardware specific bits in place, so it can be used by linaro-image-tools, flash-kernel and debian-cd.

Wednesday – May 9th

MAAS Next Steps
- Session to review and plan what are the next steps for the MAAS project, which is also missing proper ARM support for now. Great discussions on understanding all the requirements, as they will not necessarily match entirely with the usual ARM devices we have at the moment. Here the goal for ARM is to continue improving the PXE support at U-Boot (even with UEFI chainload later), and understanding what might be missing to also have IPMI support (even if not entirely provided by the hardware).
System Compositor
- Great session covering what might be the improvements and development on the graphics side for next release. Goal is to use a system compositor that would be started right at the beginning at the boot, which will then be controlled and used properly once lightdm is up (with X11). This will improve a lot the user experience on normal x86 based desktops, and luckily on ARM we’re also in a quite nice situation with the work done by Linaro helping getting the proper DRM/KMS support for the boards we support, so I hope ARM will be in a great shape here :-)
ARM Server general enhancements (for ARMv7 and perhaps v8)
- At this session we could cover what seems to be the most recurrent and problematically thing at supporting ARM servers, which is the lack of a single and supported boot method and boot loader. UEFI should be able to help on this front soon, but until then the focus will be to keep checking and making sure the current PXE implementation at u-boot works as expected (chainloading UEFI on u-boot is also another possibility Linaro is investigating). There is also the request for IPMI support, which is still unclear in general how it’ll be done generically speaking.
Integration testing for the bootloader
- As Ubuntu is also moving to the direction of continuous validating and testing all important components available, there’s the need for a proper validation of the bootloader, and the effect at the user experience while booting the system. For ARM it’s also a special case, as U-Boot is still the main bootloader used across the boards. Test case descriptions in place, and discussion will probably continue at Linaro Connect as this is also an area where we also want to help validating/testing.
ARM Server Benchmarking and Performance
- Here the Ubuntu Server Team presented how they are benchmarking and checking performance at the server level at x86, and covering what might still be needed to run and validate the ARM boards the same way. For ARM the plan is to run the same test cases on the available scenarios, and also try to get Linaro involved by making sure this is also part of the continuous validation and testing done with LAVA. Another important topic that will probably be extended at Linaro Connect is finding a way to get the power consumption data when running the test cases/benchmarks, so it can be further optimised later on.
Compiz GLES2 Handover
- Last session of the day, trying to find the missing gaps to finally get the OpenGL ES2.0 support merged at the Compiz and Unity upstream branches used by the entire Ubuntu desktop (across all archs). Following work and actions will basically be to fix the remaining and important plugins after merging the changes, and also getting a few test cases to properly validate the support at Ubuntu. Once all done, it should be merged ASAP.

These are just a few topics which I was able to participate. There are a lot of more exciting work coming on, which can all be found at http://summit.ubuntu.com/uds-q/. Remember that you’re still able to participate in a few of them tomorrow and friday, as remote access is provided for all the sessions we have.

I’m sure a lot of more exciting stuff will be discussed for ARM support until the end of this week, and at Linaro Connect, at the end of the month, we’ll be able to review and get our hands dirty as well :-)

Exciting times for ARM!

Morpheuz: Big changes!

2012-05-08T15:05:09+00:00

After 5 wonderful years working with passionate and skilled people I am now moving to São Paulo to work at Facebook’s office for Latin America. My official role will be Partner Engineer.

This doesn’t mean that I will go away from KDE, neither from Qt. It just means that I will contribute more on my free time (as it was before Nokia acquired Trolltech) than during working hours.

I am thrilled with this opportunity and I am sure that all of you who know me are also happy with this announcement.

I will be a little bit offline the next days due to my move, but I will be checking emails regularly

Ian Lawrence: Advances in mobile data collection: Nokia Data Gathering 4.04

2012-05-07T16:02:35+00:00

We have been hard at work developing and improving the Nokia Data Gathering solution, and will be launching an exciting new version in May 2012. The Nokia Data Gathering 4.04 (Dazzling Dourado) release will include bug fixes and new features on the Java mobile client, as well as a brand new Nokia Data Gathering server and user interface. We will also be launching the first Windows Phone client of Nokia Data Gathering.

Come discover and try out Nokia Data Gathering 4.04! On 16 May 2012, the Nokia Data Gathering team will be at Seton Hall University to tell you all about the new Nokia Data Gathering 4.04 server, the new Windows Phone mobile app, mobile data collection case studies from around the world, and to give you a chance to try out Nokia Data Gathering 4.04 for yourself.

Please join us!

Event	Nokia Data Gathering 4.04 workshop
Date	Wednesday, 16 May 2012
Time	09:00-12:00 AM
Place	Seton Hall University, 400 South Orange Avenue, South Orange, NJ 07079, USA
Room	Beck Rooms A, B, and C (Ground Floor of Walsh Library)

Nokia Data Gathering 4.04 is the result of a collaborative effort between Nokia, the Center for Mobile Research and Innovation at Seton Hall University and Microsoft. Learn more at http://projects.developer.nokia.com/ndg/

For more information about the workshop and/or Nokia Data Gathering, please contact kulsoom.ally 'at' nokia.com

Anselmo Lacerda S. de Melo: Qt 5 UiHelpers

2012-04-13T18:10:37+00:00

In my previous post I mentioned one of the experiments we are currently working on.
Now it is time to introduce the Qt 5 playground project called UiHelpers – Fortunately, this name will change ;)
Those who follow the Qt development mailing list already read about it, the idea about this post is to reach more people from the community in general.

Context

If you are already familiar with the structure of Qt 5 code, feel free to jump to the next section.

In Qt 4, the whole source code is in the same git repository. The traditional widgets derived from QWidget are part of the QtGui module and represent a central piece for the GUI application development.

In Qt 5, things changed. Qt code were splitted in many repositories which group Qt libraries depending on their functionalities.
Examples of repositories are qtbase, qtdeclarative, qtjsbackend, qtxmlpatterns, qtsvg, qttools, etc.
As you may know, QtQuick 2 became a first class citizen in Qt 5, which means qtdeclarative is one of the main repositories, house of QtQuick and QtQml classes.

But, there are classes that provide the base for QtQuick 2 work properly – they are part of the qtbase repo. There you find QtCore,
QtGui, QtNetwork, QtOpenGL, QtPlatformSupport, QtXml, QtWidgets, etc.

Wait: What’s the difference between QtGui and QtWidgets?

As part of the changes of Qt 5, QtGui is now focused in the basic infrastructure for GUI work on different platforms. For example, it is the place for the platform abstraction layer, window management, image handling, etc.

QtWidgets, as you can conclude based on its name, contains the children of QWidget: QPushButton, QSpinBox, QMainWindow, etc. Also, there are non-widgets classes in QtWidgets, some of them because they are internally too coupled to QWidget classes.

Origins

In this thread it was mentioned the case of QUndoStack, QUndoCommand that are in QtWidgets but can be useful in other contexts where developers do not want to link against QtWidgets just because of this kind of helper classes.
Later, in the change 15857 (with the help of ogoffart), we discussed about moving QUndo* out of QtWidgets. First, we moved these classes to QtGui what as not considered correct given the new motivation of this lib. Then, another idea was to move the classes to a new lib, inside qtbase. It was not considered ideal and the decision was to create a separated repo for these classes, leaving QtWidgets untouched (since the Qt community do not want to introduce new bugs in this lib).

Current status

We set a wiki page where it is possible to check the classes made available out of QtWidgets. Also, there are instructions about where to download the source code, build, test and contact us.

Besides moving classes out of QtWidgets, some prototypes of convenience APIs were created for QtQuick 2 developers and we are very interested in receiving feedback about them. It’s worth mentioning the QML API for UndoStack and the CompletionModel which is based on the internals of QCompleter and was made available in a QML friendly way.

The following example shows a possible use case for the CompletionModel. Improving the models supported by sourceModel is part of the ToDo list. The ListView is going to show the updated list that currently satisfies what is written in the TextInput.

TextInput {
    id: input
}

CompletionModel {
    id: completionModel
    sourceModel: ["Ascension Island", "Andorra", "Afghanistan"]
    completionPrefix: input.text
    caseSensitivity: Qt.CaseInsensitive
}

ListView {
    model: completionModel
    delegate: Text { text: modelData }
}

The two examples bellow show the UndoStack and two items that represent commands. The idea of UndoPropertyCommand is to enable track the changes of some properties. In the example, we are going to track x and y when myRect is moved.

UndoStack {
    id: stack
    undoLimit: 5
}

UndoPropertyCommand {
    id: moveCommand
    properties: ["x", "y"]
}

Rectangle {
    id: myRect

    (...)

    MouseArea {
        (...)
        drag.target: parent
        onPressed: stack.push(moveCommand, myRect);
    }
}

UndoCommand is another element used to represent commands. The difference here is that it allows the customization of the actions onUndo and onRedo the command.

UndoCommand {
    id: aCommand

    onUndo: doAThing(target);
    onRedo: undoAThing(target);
}

Button {
    onClicked: stack.push(aCommand, target);
}

Future

Maybe one day this playground project gets promoted and become a Qt Add-on, but there is no expectation it will happen before 5.0. In the meanwhile, there is a ToDo list with some topics we consider important to handle, like better examples, more tests, docs, etc.

We are looking forward receiving more opinions about this work and, as always, contributors are welcome!

Bruno Abinader: Moving out! :)

Bruno de Oliveira Abinader — 2012-04-11T11:12:21+00:00

Hello my fellow readers :)

First of all, this is not a "goodbye" message! It's time to move my blog to a cool domain, which is commonly used by me fellow colleagues Igor Oliveira and Adenilson Cavalcanti. All of my previous posts were moved to there, so don't worry (I mean the CMake tutorials!). So, the new blog link is below:

http://bruno.abinader.com.br

Thank you BlogSpot for hosting this blog for 3 years without problems, and I hope to see you there! :)

Anselmo Lacerda S. de Melo: Working in the open again

2012-03-27T12:40:01+00:00

Long time since my last post (it’s sad to realize how often I start posts writing this).
Since last year’s Qt Contributors’ Summit, our team at INdT worked on a closed source project – the main reason of the low activity here. At least, I could write some posts for the Qt Labs Blog Brazil, most of them introducing basic concepts about Qt programming and the Qt ecossystem (buzzword++).
The good part is: This project we were working on was developed using Qt and QML for a desktop application. Besides promoting Qt among Brazilian developers and companies, this project helped us to think about improvements for future releases of QtQuick aiming desktop applications. I am looking forward to see a release of this product soon! (However, when it is going to happen is a client decision).

Then, after the end of this project, I finally managed to contribute to Qt5! :-)

Also, in the end of the last year some coworkers started a kind of pet project: It’s called Iceberg, a fork of the Icecream Monitor.

“Why a fork?” Here we use icecream for distributed compilation, many colleagues used to complain about icemon depending on kdelibs. In fact, the code did not use the additional features compared to the equivalent Qt classes. Then, after hugopl started the project, we fixed some issues and added new stuff. The code is here.

I am writing another post to introduce one of our current research projects, which is also part of the proposal I sent to Akademy 2012.

Ricardo Salveti: ARM Porting Jam this Friday!

rsalveti — 2012-03-02T07:10:26+00:00

For those following the development of the next Ubuntu release (12.04 – Precise Pangolin), you all know that we’re quite close to the release date already, and to make sure Precise rocks since day 0, we all need to work hard to get most of the bugs sorted out during the next few weeks.

At Linaro, the Linaro Developer Platform team will be organizing an ARM porting Jam this Friday, with the goal of getting all developers interested in fixing and working on bugs and portability issues related with the Ubuntu ARM port (mostly issues with ARMHF at the moment).

The idea of having the Porting Jam at Friday is to have it as a joint effort with Ubuntu’s Fix Friday and Ubuntu Global Jam, so expect quite a few other developers helping improving Ubuntu as well!

It’s quite easy to participate:

Check the current bug list for ARM at http://people.linaro.org/~rsalveti/arm-porting-queue/arm-porting-queue-report.html and FTBFS list from http://qa.ubuntuwire.com/ftbfs/
Look at http://developer.ubuntu.com/packaging/html/ for more information about packaging, and how to provide patches and fixes for our current packages
Join #linaro and/or #ubuntu-motu on irc.freenode.net
Grab a bug, debug, ask questions and update the bug report with your findings
Get involved with other developers

Remember that for ARM this release will be a quite huge milestone, as it’ll be the first LTS release supporting ARM, besides delivering support for ARM servers and ARMHF as default, so let’s make sure it rocks!

Looking forward for a great porting Jam!

Happy bug fixing!

Ricardo Salveti: Ubuntu TV UI at Pandaboard, and next steps

rsalveti — 2012-01-16T06:59:56+00:00

Yesterday Canonical announced the first UI concept for the Ubuntu TV. Together with the announcement, the first code drop was released, so we could read and understand better the technologies used, and how this will behave on an ARM environment, mostly at a Pandaboard (that we already have OpenGL ES 2 and video decode working).

Getting Ubuntu TV to work

If are still using Oneiric, you can just follow the guide presented at https://wiki.ubuntu.com/UbuntuTV/Contributing, where you’ll find all needed steps to try Ubuntu TV at your machine.

As it’s quite close with Unity 2D (similar code base), and also based on Qt, I decided to follow the steps described at wiki page and see if it should work correctly.

First issue we found with Qt, was that it wasn’t rendering at full screen when using with latest PowerVR SGX drivers, so any application you wanted to use with Qt Opengl would just show itself on a small part of the screen. Luckily TI (Nicolas Dechesne and Xavier Boudet) quickly provided me a new release of the driver, fixing this issue (version that should be around later today at the Linaro Overlay), so I could continue my journey :-)

Next problem was that Qt was enabling brokenTexSubImage and brokenFBOReadBack for the SGX drivers based on the old versions available for Beagle, and seems this is not needed anymore with the current version available at Pandaboard (still to be reviewed with TI, so a proper solution can be forwarded to Qt).

Code removed, patch applied and package built (after many hours), and I was finally able to successfully open the Ubuntu TV interface at my Panda :-)

UI Navigation on a Pandaboard, with Qt and OpenGL ES2.0

Running Ubuntu TV is quite simple if you’re already running the Unity 2D interface. All you need to do is to make sure you kill all unity-2d components and that you’re running metacity without composite enabled. Other than that you just run ”unity-2d-shell -opengl” and voilà ;-)

Here’s a video of the current interface running on my Panda:

As you can see from the video, I didn’t actually play any video, and that’s because currently we’re lacking a generic texture handler for OpenGL ES with Gstreamer at Qtmobility (there’s only one available, but specifically for Meego). Once that’s fixed, the video playback should behave similarly as with XBMC (but with less hacks, as it’s a native GST backend).

Next steps, enabling proper video decode

Looking at what would be needed to finally be able to play the videos, and to make it something useful at your Pandaboard, the first thing is that we need to improve Qtmobility to have a more generic (but unfortunately still specific to Omap) way handle texture streaming with Gstreamer and OpenGL ES. Rob Clark added a similar functionality at XBMC, creating support for ”eglImage”, so we just need to port the work and make sure it works properly with Qtmobility.

Once that’s ported, the video should be streamed as a texture at the video surface, making it also work transparently with QML (the way it’s done with Ubuntu TV).

If you know Qt and Gstreamer, and also want to help getting it to work properly on your panda, here follows a few resources:

Rob’s implementation of eglImage at XBMC: https://github.com/robclark/xbmc/commits/gstreamer-eglimg
Qtmobility support for Meego and OpenGLES: http://qt.gitorious.org/qt-mobility/qt-mobility/blobs/master/plugins/multimedia/gstreamer/qgstreamergltexturerenderer.cpp
IRC: #linaro and #ubuntu-tv at Freenode, look for robclark or rsalveti

As soon video decoding is working properly, a new blog post should be around explaining the details and how to reproduce it at your own Panda with Ubuntu LEB :-)

Cheers!

Ricardo Salveti: Ubuntu TV fully accelerated on a Pandaboard with Ubuntu LEB

rsalveti — 2012-01-16T06:56:11+00:00

As described on my previous post about Ubuntu TV support on a Pandaboard, we were still missing proper support for texture streaming on a Pandaboard, to have the video playback also working and fully accelerated.

This weekend Rob Clark managed to create the first version of the TI’s specific eglImage support at Qtmobility, posting the code at his gitorious account, and for the first time we’re fully able to use Ubuntu TV on a ARM device, using a Pandaboard.

Demo video with the Ubuntu TV UI (accelerated with Qt and OpenGL ES 2.0) and with video decode support of 720p and 1080p:

The code support for TI’s eglImage still needs a few clean-ups, but we hope to be able to push the support at Ubuntu in the following weeks (make it good enough to try at least a package patch).

For people wanting to try it out, a few packages are already available at Linaro’s Overlay PPA, and the remaining ones should be available later today (Qt and Qtmobility), so people can easily run it with our images.

Hope you enjoy, and we’ll make sure we’re always working on keeping and improving the current support, so Ubuntu TV also rocks with ARM :-)

Cheers!

Ricardo Salveti: HW video decode and XBMC support on a Pandaboard with Ubuntu LEB

rsalveti — 2012-01-16T02:11:41+00:00

Part of the effort we spent during the Linaro 11.12 cycle was to try to enable at Pandaboard not only hardware graphics support (GLES with PVR SGX), but also hardware accelerated video decode, as TI had released all needed userspace to be used at Ubuntu Oneiric (11.10) release.

Unfortunately it didn’t just work with our images because at that time we were using a newer kernel already, based on the 3.1 series that is maintained by the Linaro TI Landing Team. Bug 880840 has all the details.

Luckily Sebastien Jan (from TI) was able to find the root cause of the problem, that was causing so much frame drops that was making the video playback basically unusable. The problem was related with PM support at omap’s hwspinlock implementation, as you can check at this link.

Kernel fix properly integrated and available at the Overlay PPA used by our Linaro Ubuntu Evaluation Build images, and finally able to have a similar user experience as was expected when TI delivered the user space components at their own PPA.

If you want to try it by yourself, just be sure you’re using at least linux-image-3.1.1-6-linaro-lt-omap at your board (all hwpacks >= 20110105 should have it included by default).

Playing videos with HW decode acceleration

Since today you’ll also easily find all the needed packages to enable HW video decode acceleration at our images (Pandaboard only at the moment, more boards coming soon). We just included and copied all needed packages from the TI PPA, so you don’t even need to enable it when installing the additional packages.

Installing the extra packages for video decode at your Pandaboard:

Grab the latest Pandaboard hwpack (lt-panda-x11-base-oneiric) and Ubuntu Desktop image from http://snapshots.linaro.org/oneiric (as example I used hwpack_linaro-lt-panda-x11-base_20120106-0_armel_supported.tar.gz and linaro-o-ubuntu-desktop-tar-20120105-0.tar.gz)
Create a Ubuntu LEB pandaboard image on a SD card, following the instructions described at https://wiki.linaro.org/Platform/DevPlatform/Ubuntu/ImageInstallation
Boot the card and install the ubuntu-omap4-extras-multimedia package: $ sudo apt-get install ubuntu-omap4-extras-multimedia
Reboot your pandaboard
Play a video with any video player that’s compatible with Gstreamer (e.g. Totem)

In the future we should also have this completely integrated at the hwpack itself, but unfortunately this is not possible at the moment without increasing the image size too much.

XBMC support

Another awesome thing we worked during previous cycle (11.12) was to make an XBMC version available that would use both GLES and Gstreamer, so it could also be used with a Pandaboard. Avik Sil did a great work making it all work with our images, and we were finally able to have XBMC 11 Beta (Eden) available at our Overlay PPA.

For proper support for Gstreamer Rob Clark did an awesome work improving the current patches, and also improving the support quite a bit. At our package you’ll find all latest patches available from Rob, from his current development tree.

To start using XBMC with the Ubuntu LEB image at your Pandaboard, you just need to install the xbmc package, with $ sudo apt-get install xbmc. For best user experience, please use the XBMC session available at LightDM (just log-out the default session and select XBMC instead). This will work a lot better because then there will be no other window manager or compositor taking extra resources from your board.

We also hope to deliver a set-top box image by the end of the current cycle (12.01), that will have XBMC installed by default. Please check the blueprint https://blueprints.launchpad.net/linaro-ubuntu/+spec/create-a-set-top-box-leb-image if you want to follow the progress of it.

Bugs and Issues

Unfortunately not everything is working perfectly at the moment, and issues with the Gstreamer and hw video decode support on Pandaboard are expected. The most annoying one that’s currently affecting XBMC is the issues with seek, as sometimes the video goes faster than the audio, and then it stops for a while until it’s in sync again. We hope to get this fixed soon, but that depends a bit of how much time Rob can spend on it.

In case of any other bug while trying to get video decode to work on your Pandaboard, don’t hesitate to open a bug at https://bugs.launchpad.net/linaro-ubuntu/+filebug or ping aviksil, robclark or rsalveti at #linaro on freenode.

Cheers!

Update: Check bug https://bugs.launchpad.net/linaro-ubuntu/+bug/915456 for the video hanging issue. Without polling XBMC should now play most videos just fine.

Update 2: XBMC-ready image already available at http://snapshots.linaro.org/oneiric/linaro-o-linarotv-xbmc/, just be sure to flash with http://snapshots.linaro.org/oneiric/lt-panda-x11-base-oneiric/.

Update 3: There’s a mem leak at the gst decode codec, check bug https://bugs.launchpad.net/ubuntu-omap4-extras-multimedia/+bug/915768 for progress on that.

Ademar de Souza Reis Jr.: Retrospectiva 2011, planos pra 2012

2011-12-31T21:36:29+00:00

Pra não perder a tradição (desde 2006), hora do post de fim de ano: relembrar 2011 e planejar um pouco 2012.

2011 foi um ano excelente pra mim. Um ano pra ser lembrado. Começou com novidades ruins no trabalho no INdT/Nokia: “the burning platform”, a aliança com a Microsoft e o futuro incerto. Pra aliviar um pouco, vieram as férias, uma excelente viagem à Europa com a Viviane: 35 dias intensos, 11 países (França, Inglaterra, Holanda, Alemanha, Russia, Polônia, República Checa, Áustria, Itália e Vaticano — sim, o Vaticano conta!). Ótimas experiências com couchsurfing, muitos kms de estradas, trens, aeroportos… Viagem intensa mas sem nenhum contratempo.

Mosaico com algumas fotos da viagem à Europa

Mais pro final do ano vieram duas grandes e excelentes novidades, quase que simultaneamente:

Estávamos começando a fazer nossos planos pra ter um bebê quando veio a surpresa, mais do que bem vinda: a Viviane estava grávida. Estamos super empolgados e felizes. Em 2012 (abril/maio) deve chegar a Sofia, nossa primeira filha. Foi uma correria pela coincidência com as férias e aproveitamos pra visitar nossos pais (RS e PR). Também fizemos uma viagem rápida aos EUA pra aproveitar nossas últimas férias sozinhos e fazer o enxoval do bebê. Fomos pra Miami e Orlando, aliando compras com alguns passeios.

Mosaico de fotos da viagem à Miami e Orlando

A segunda grande novidade foi a mudança no trabalho: após um longo e criterioso processo de entrevistas (~3 meses), consegui uma vaga na empresa que mais admiro, pra trabalhar numa área na qual eu sempre tive muito entusiasmo mas pouca oportunidade: entrei pra Red Hat como gerente técnico no time de virtualização (kvm e qemu – kernel e afins), um time cheio de grandes talentos com foco nos projetos upstream, um grande desafio. Na Red Hat me senti em casa, como se eu tivesse voltado aos tempos de Conectiva: revi bons e velhos amigos e me identifiquei de imediato com os novos colegas. E no último mês do ano ainda veio mais uma novidade: consegui ser aprovado no processo de seleção do Mestrado em Computação da UFPE/Cin com uma proposta na área de virtualização.

A revisão dos objetivos de 2011 ficou assim:

~~Visitar dois países que eu ainda não conheça;~~ OK – o plano original era ir pra China e Índia, mas os 9 novos países europeus cumprem o objetivo com folga;
Atualizar a seção de fotos do site; FAIL – sem comentários;
~~Visitar nossos familiares e amigos no sul do Brasil;~~ OK – dava pra fazer melhor, mas Viviane foi pro RS e fomos juntos pra Umuarama – PR;
Começar a estudar um novo idioma, junto com a Viviane; FAIL
~~Estudar (formalmente ou não), pois evoluí menos do que esperava em 2010;~~ OK – foquei um pouco nos estudos no final do ano, 2012 com certeza será melhor;
~~Melhorar minha atividade nas “redes sociais”, reforçar antigas amizades;~~ OK – acho que posso melhorar, mas estou no caminho certo…
Comprar algum tipo de tablet, preferivelmente um que sirva como ebook-reader; FAIL – ficou pra 2012, um Kindle está fazendo muita falta.

2012 será muito intenso e com desafios sem tamanho. Mas já começo realizando o tradicional objetivo de conhecer um novo país todo ano: em janeiro vou pra Israel a trabalho. Dessa vez não farei uma lista de planos específicos. Tenho apenas três grandes objetivos, em ordem de prioridade:

Ser um bom pai pra Sofia e um bom marido pra Viviane nessa nova fase de nossas vidas;
Ser um bom profissional na Red Hat, cumprindo com as expectativas e superando minhas limitações;
Conseguir manter um bom ritmo no primeiro ano do mestrado na UFPE/Cin.

É isso aí, um ótimo 2012 pra todos!

Link Permanente | 3 commentários
Post tags: retrospectiva

Ricardo Salveti: UDS-P/Linaro Connect Q4.11 and 11.11 cycle

rsalveti — 2011-12-05T16:54:33+00:00

During the end of October and beginning of November we had the last Linaro Connect for the year. This time we also had it together with the Ubuntu Developer Summit, giving us the opportunity to better discuss the roadmap with both Linaro and the Ubuntu team.

From the Developer Platform team perspective, we had a quite nice week, with demos happening at Monday and Friday (showing people what we’ve been working on), and also sharing some great news with the Ubuntu team, now that Mark Shuttleworth announced that Ubuntu will go to Tablets, TVs and Phones (and ARM for sure will be a huge part of that).

Some nice links and videos of what happened during that week (related with our team):
* Sessions related with the Developer Platform Team (Ubuntu)
* Linaro Demo: Ubuntu Unity with OpenGL ES on Pandaboard
* Linaro Developer Platform Tech Lead Ricardo Salveti Interview at Linaro Connect
* Linaro Connect Q4.11 – Ubuntu LEB tutorial
* Linaro Connect Q4.11 – Interview with Marcin Juszkiewicz

Linaro 11.11 Release

Another quite good achievement for us during November was the 11.11 release.

During this release we had a quite a few great highlights, including some that we were planning for quite a while already:
* Ability to cross build Firefox using Multiarch
* OMAP4 SPL USB Booting, enabling USB boot at Pandaboard
* ARM DS-5 support for the 5.8 release
* CI Builds for Linaro GCC both for cross and native
* And a lot of bug fixes

Now it’s time to get ready to develop the blueprints we’re planning for 11.12, to also make December another great and solid month :-) (will do another post about the 11.12 planning later this one).

Morpheuz: Back to life!

2011-11-21T19:08:59+00:00

I am back from my honeymoon and I’m starting to organize my life again!

Meanwhile, we finally released a game that I’m really proud that is called “Incredible Circus”. It’s the kind of game that is really addictive :). In three weeks we achieved 200k downloads on Nokia Store and today is available for the N9. It’s also a really good showcase for Qt. If you have a Symbian^3 or MeeGo device, just follow the link to try the game.

You can also check it out on the video below.

Apart from that, I’m updating my machine and as soon as I have a developer environment again I’ll start hacking on Plasma again. Probably fixing some bugs, right Aaron?

Danilo Cesar: Prague, Telepathy and HTML5 IM Client

2011-11-03T12:37:40+00:00

Hey guys!

Last week I’ve attended to GStreamerConf and LinuxConf conferences in Prague, and Siraj and I had the honor of having our current project for Collabora exposed on Collabora’s booth. It shared space with two others amazing Collabora‘s demos:

Media Explorer running a Telepathy plug-in, which adds IM and Audio/Video capabilities to the system.
HTML 5 client of a video editor, using GStreamer Editing Services on top of GtkWebkit.

A little bit about Telepathy Webkit Client

During the last few weeks I’ve been working on a Telepathy client, a very special one, written almost only with HTML 5 and Javascript, on top of GTK WebKit and Seed.
The goal of this project was writing a Telepathy based IM client, to run on top of WebKit, and it should be able do make and receive audio and video calls.

GObject introspection magic

The first question to be answered was “How access the Telepathy methods from Javascript?”

And the answer was: GIR and Seed.

As all Gnome developers know, GObject introspection (GIR) allows us to access a GObject written in C from a scriptable language, such as Javascript. So that’s part of the answer.

And Seed binds those introspected objects on WebKit JavascriptCore, which is pretty much what we wanted to do.

Following a few screen-shots

Contact List

Our first goal was to show the Telepathy’s roaster list on a HTML5 window. For the widgets, we choose to use joApp, which is a quite nice and stable UI library for HTML 5 applications.

Chat Window

What’s a IM client without a Chat window, right?!

The “tpwebkit” application with an opened text channel between two users.

Receiving a call

Telepathy WebKit application acting as a Telepathy approver for audio and video calls

That’s me, from the point of view of my test machine…

The real thing running

If you want to see the real thing running, you can watch this video showing the application during the early development state.

Conclusions and future work

As we can see from now, Telepathy is ready to be use on native and HTML applications.
I really would like to see the current implementation integrated with the bleeding edge RTC API, which some guys are pushing to WebKit. That would be something really awesome.

If you want to try it, you can get the code here and play with it. You’re also going to need a few special changes on Seed package. Check the linuxconf branch here.
Or, if you use Ubuntu, you can add Xavier’s PPA and install the tpwebkit package.

Ian Lawrence: Tangaza or 'announce' in Swahili

2011-10-14T13:54:30+00:00

Despite the rapid growth of mobile data and mobile internet, many (most) people in developing countries do not have access to mobile data. For example in Africa, there are around 45 basic mobile subscriptions but only 2 data subscriptions per 100 inhabitants. Peer to peer communication, voice calls, SMS and 'beeping' are the standard communication tools and techniques currently in use.

The success of Twitter and other group communication tools has been undeniable and the development and introduction of mobile app based messaging systems remains active. But would it be possible to design a group messaging system for people who only have access to a basic feature phone with standard GSM features available on every mobile phone?

Tangaza is a mobile phone based group messaging system targeted at moderately low income users. Through the use of missed calls, a basic SMS command set and concise touch tone menu, Tangaza is designed to enable affordable spoken communication among flexible groups of users. Developed in collaboration between Nokia US and African research units, Nokia Tangaza was piloted in Kenya in 2009 and a research paper (.pdf) on this pilot and on Tangaza's design decisions was presented at the 1st Annual Symposium on Computing for Development in 2010. The software has now been open sourced under the AGPL and I have just submitted an ITP to the Debian Bug Tracker. We aim to have Tangaza in the next Debian release and the Debian VOIP team are helping with this effort.

Tangaza runs on Linux and you will need to have

Perl 5
Python 2.6
MySQL 5.0
Django 1.2
Kannel 1.4.3
Asterisk 1.6.2

installed on your Debian based system if you want to build Tangaza before it reaches the main repositories . You need to

git clone https://github.com/tangaza/Tangaza

git checkout -b upstream --track origin/master

(The '--track' option alters your .git/config file and adds a [branch "upstream"] section telling Git where you fetched it from. That means you can later just say "git pull" and you will get both the 'master' and the 'upstream' repository merged into your repository automatically.)

git checkout master

git-buildpackage --git-ignore-new --git-builder=debuild -i.git -I.git -us -uc

and install the resulting .deb package

During the last development cycle we built Tangaza a web enabled administrative interface in Django where the groups, members and organization information can be modified and the messages (called Tangazo's) can be listened to We also wrote some documentation about the API calls we make available through RPC for developers who want to build applications on top of Tangaza.

There is some interest in using Tangaza as an agricultural early warning system in East Africa. A Humanitarian Innovation Fund expression of interest passed through to a call for a full proposal which is pretty exciting both in terms of potential impact and also for the new features which can be developed with increased funding

Morpheuz: It has been a while

2011-09-14T12:24:44+00:00

It has been a while since my last post and a lot happened in the meantime. Just to point a few (in no special order):

11th February for Nokia;
Tokamak;
GSoC;
Qt Contributor Summit;
Qt Open Governance announced;
Desktop Summit;
Release of N9;
Google acquired Motorola;
Plasma Active;
Netflix in Brazil;
Use of Tiny Tiny RSS (my share feed)
…(many other stuff!)

The sad part though is that I haven’t been able to contribute much code to KDE lately. Next month I’m getting married and as you can probably guess, there is a lot of planning going on (and we can’t forget the famous “bridezilla” effect that usually happens with girls before weddings

On the bright side I’ve been working on awesome projects, trying to eat our own dog food (mainly QML). With these projects we are feeling the pieces of Qt/QML that needs some love and we plan that for the next year we will be able to help the Qt Project to solve these ~~small~~ issues that you can only find when you develop real world applications and not developer/designer-wanna-be demos

IMHO that’s one of the skills that is hard to find today on framework developers: most of them didn’t contribute much to KDE in the last years nor developed real applications. Because of this sometimes it gets harder for them to understand a use case or a “complain” about the technology they develop. I hope that with open governance we can get more help about this (and also use more the rule of the “3 examples” before adding new APIs).

I’ve been following closely the development of Plasma Active and the rest of KDE. I still have one item on my TODO list (related to one of our scripts that get the content of qml files that need translation) that I promised to tsdgeos that I would fix ASAP but I just didn’t have the time unfortunately . November seems the month that I will be able to get back to KDE development (after the honeymoon!!).

Well, I think that’s it. I just wanted to make a simple blog post and update my KDE friends

Adenilson Cavalcanti: CellarDoor in German plus Debian package

savago — 2011-08-17T01:23:10+00:00

Dear friends

Some quick updates on CellarDoor are due. Since last week, it has debian packaging support in the buildsystem, contributed by Milton Soares.

Simply run ‘make deb’ and you are set: a nice debian package is generated at end.

Next, Adriano Cavalcanti contributed another translation to CellarDoor, German:

Thanks to the fact that German has some bigger than average words, it will require some adjustment in some parts of the UI, but so far so good! As soon I finish some missing features, I’m planning to make a ‘non-hackers’ release.

Currently CellarDoor is available in:

English
Brazilian Portuguese
Italian
German

Do you known French or Spanish?
:-)

Adenilson Cavalcanti: CellarDoor got its second translation: Italian!

savago — 2011-08-08T14:16:35+00:00

Friends

Just 1 day after blogging about my pet project CellarDoor, I was contacted by Francesco Frassinelli asking about how to make translations.

The result in the next day can be checked below:

Of course there are some places in the UI that we will need to make minor adjustments, but it certainly is progressing really fast (except for minor UI code glitches, the Italian translation is done).

Do you speak German or French? Want to contribute to OSS/Freesoftware? Let me known.
:-)

Adenilson Cavalcanti: Presenting CellarDoor

savago — 2011-08-06T14:01:43+00:00

Last January I was talking with my friend Wilson Prata about applications. We have partnered a couple of years ago when creating amora (which by the way needs some care urgently, but this is another matter) and he presented to me a concept for a nice and cool new app: CellarDoor.

What is CellaDoor? Well, it is a wine app. Have you ever started a conversation with someone that also appreciate wines and then failed when trying to remember the name of a good wine you tasted a couple of months ago?

Or what about easily exchanging a wine recommendation? Trying to keep notes in a restaurant napkin is not really much effective.

Trying to address this and other user cases, I started to work in the mockups that he had at time. A great UX designer once taught me the value of close cooperation between programmers X designers to create something that looks good and performs well (and I honestly believe that the best way to create an app is to have at least one designer aboard since the very beginning). This is a different approach than the traditional hack-hack-hack then later ask a designer for some cute icons which is, with some exceptions, the rule in OSS/Freesoftware world.

Unfortunately, I lacked enough free time to progress quickly, but I believe that the app has progressed well enough for a public release. You can checkout the project’s webpage.

So, what are the current features? Here they are:

good looking UI with nice usability
create a wine card with basic information (year, name, vineyard, etc)
filter by wine type (e.g. red, white, other)
persistence of data in a sqlite database (which by the way you can edit in the desktop)
Supported platforms: Linux, OSX, Windows 7, Symbian 5th and 3 (e.g N8, N5800, etc)
snap a photo of the bottle/cork in supported platforms (Linux, Symbian). Persistence of this photo is my todo list (hey, I didn’t say that the app is completely done!).

How it looks like? Check it (running in OSX 10.6):

== What about the technology? ==

The magic to support such varying OSes (Linux, Window, OSX, Symbian), form factors (Mobile, Netbook, Desktop) and resolutions (from 360×640, ranging in 1024×800 and up) is that I used Qt/QML for the UI. At time that I started, Qt Components were not ready yet, so I got to implement my own (probably buggy) widgets like a combobox (based in joint work with my other friend Ricardo Sato) and a calendar widget (coded together with my friend Igor Trindade), this last widget deserves to be pictured bellow:

== What is next? ==

A long backlog indeed waits in the issue tracker of the project. Between those, I believe that next steps should be:

Packaging: for symbian I have it done (self signed), Window7 I played with WiX to create an installer but I need help for testing. OSX packaging (.dmg) and rpm/deb help would be greatly appreciated;
Translation: it currently supports English and Brazilian Portuguese. I believe it would be cool to support other languages (French, Italian, German);
Coding: there are several features that would be awesome to have, if you are interested, let me known.

So, since ATM there aren’t public packages yet, I guess this is a ‘hacker’s only release’. What you need to compile it is Qt 4.7.x and if in Linux, Qt Mobility 1.1.

Concerning the license, it is good and old GPL v2 code with CC non-commercial for the artwork.

Adenilson Cavalcanti: Long time no see you

savago — 2011-08-05T22:44:29+00:00

It is being almost one year without posting in my blog… What else to say, besides I have being busy? Since my last post, I have:

a) Trying to be a not so lousy father;

b) Led 2 teams in 2 different projects at same time at OpenBossa (both non-oss, targeting Symbian and Maemo respectively);

c) Visited Boston, Brussels, Miami, London, Bristol, Jericoacoara;

d) Improving my old motorcycle with new accessories;

e) Bought a Brazilian off road “car” and travelled through the country side;

f) Moved to a new apartment, with space for a small office;

g) Since just motorcycling is not enough, tried some new things:

h) After learning QML, decided to write a new pet project (details in the next post!)

i) Reviewing and merging several patches in libgcal (news soon!)

And probably some other stuff that I plainly forgot. Now, I’m planning to post once in a while to not let my blog to collect dust… after all, like I said way back in September 2009: “… having a half-alive, half-dead, zoombie-like blog is worst than having no blog at all.”.

Ricardo Salveti: launchpadlib: creating a changelog for a PPA

rsalveti — 2011-07-19T03:34:58+00:00

As at Linaro we usually work with many PPAs over the releases, there was a need to generate a proper changelog for a PPA, in a way we could know what packages got changed before doing the release.

At first I thought I could just parse the repository metadata (as a PPA is nothing more than a debian repository), but then I realized I could just use the awesome (yes, *awesome*) launchpadlib, if it had a way to get the data I needed.

So I called the launchpadlib master I know (Ursinha), and in 15 minutes we saw that we could use it to parse the “.changes” file, and from there get the data I needed. As Launchpad stores the PPA packages publishing history, it’s quite easy to get all the changes over period of time.

A few minutes later (after also noticing that there’s a python-debian module to parse the changes file), I created the first version of the generate-ppa-changelog.py script, that does exactly what I needed, and with just a few python lines :-)

Here’s the link: http://bazaar.launchpad.net/~rsalveti/+junk/launchpad/view/head:/generate-ppa-changelog.py.

Usage:

rsalveti@evatp:~/projects/launchpad/scripts$ python generate-ppa-changelog.py -h
usage: generate-ppa-changelog.py [-h] [-d YYYYMMDD] [-s SERIES] -t TEAM
[-p PPA] [–version]

Print Changelog for a PPA

optional arguments:
-h, –help show this help message and exit
-d YYYYMMDD, –date YYYYMMDD
start date to probe for changes
-s SERIES, –series SERIES
ubuntu series to look for changes (default: natty)
-t TEAM, –team TEAM launchpad team that owns the PPA
-p PPA, –ppa PPA ppa name to probe the changelog (default: first PPA)
–version show program’s version number and exit

If no argument is given, it will probe all the changes for the default series.

Output Example:

rsalveti@evatp:~/projects/launchpad/scripts$ python generate-ppa-changelog.py -t linaro-maintainers -p overlay -s natty -d 20110701
Changelog for linaro-maintainers’s overlay PPA (series natty) since 2011-07-01 00:00:00

base-files (5.0.0ubuntu28linaro3) natty; urgency=low

* Updating Linaro LEB version to 11.07 (development branch)

— Ricardo Salveti de Araujo Fri, 15 Jul 2011 04:19:40 -0300

libjpeg-turbo (1.1.1-1inaro2) natty; urgency=low

* release
* add timestamp code to cjpeg.c
* default cjpeg.c and djpeg.c timestamp code to off

— Tom Gall Mon, 11 Jul 2011 20:32:23 +0000

linaro-meta (017) natty; urgency=low

* Refreshed dependencies
* Added libjpeg-turbo62 to linaro-alip, linaro-graphical-engineering,
linaro-multimedia-engineering, linaro-ubuntu-desktop

— Tom Gall Wed, 13 Jul 2011 15:41:29 +0000

u-boot-linaro (2011.07.1-0ubuntu1~natty1) natty; urgency=low

* New upstream 2011.07.1 which includes
– PXE FDT fix from previous ubuntu release so patch has been removed
– Added missing PXE env vars (LP: #808815)
– Generated unique usbethaddr (LP: #809015)
– Modify pxe command to look for usbethaddr if ethaddr is NULL

— John Rigby Wed, 13 Jul 2011 23:31:39 -0600

x-loader (1.5.1+git20110715+fca7cd2-1ubuntu1~natty1) natty; urgency=low

* New upstream release
– several fixes for panda and igep
– OMAP3 code cleanup
– Beagle Rev C5 support
– Support for IGEPv3 board

— Ricardo Salveti de Araujo Fri, 15 Jul 2011 05:55:09 -0300

x-loader (1.5.0+git20110714+cdc887b-1ubuntu1~natty1) natty; urgency=low

* New upstream release
– Adding support for IGEPv3 board
* debian/patches/01-Beagle-Rev-C5-support.patch:
– Adding support for the new Beagle C5

— Ricardo Salveti de Araujo Fri, 15 Jul 2011 02:36:58 -0300

And now we can just generate the proper changelog anytime we want to do a call for testing or a Linaro Ubuntu release :-)

Ricardo Salveti: Net booting with TFTP and PXE with Pandaboard

rsalveti — 2011-07-11T09:38:44+00:00

Over the past month I’ve being working with John Rigby to integrate the SMSC95XX and OMAP4 EHCI patches into Linaro U-Boot, so we could deliver the network booting feature for people using Pandaboards.

Those patches are published at the U-Boot mailing list, but still as a working in progress. While we work helping the original developers to get the patches accepted upstream, we also want to deliver the functionality for our users, so all those patches are now integrated at the Linaro U-Boot tree.

You can check the patches by going at http://git.linaro.org/gitweb?p=boot/u-boot-linaro-stable.git;a=shortlog.

Testing with Pandaboard

To make it work properly, besides using Linaro U-Boot you’ll also need to use the upstream X-Loader tree, with one additional patch that’s not yet merged. You can clone the upstream tree from http://gitorious.org/x-loader/x-loader, then just apply the patch http://people.canonical.com/~rsalveti/pxe/0001-omap4-pandaboard-ehci-fref_clkout-per-board-revision.patch and build for the Pandaboard target.

If you just want to test without building your own X-Loader and U-Boot, you can just grab both files from http://people.canonical.com/~rsalveti:

X-Loader: http://people.canonical.com/~rsalveti/pxe/MLO
U-Boot: http://people.canonical.com/~rsalveti/pxe/3/u-boot.bin

Building your TFTP + DHCP server for PXE

To build your TFTP + DCHP server just follow the instructions described at https://help.ubuntu.com/community/Desktop/PXE. Don’t worry about the ‘filename “pxelinux.0″;’ line at the dhcpd.conf file, you can remove it.

Then just create your PXE config file at the right place:

$ cat /tftpboot/pxelinux.cfg/0A2A2B0A
default panda-natty
prompt 0
timeout 3

label panda-natty
kernel panda/uImage
append console=ttyO2,115200n8 root=/dev/mmcblk0p2 ro fixrtc vram=48M omapfb.vram=0:24M mem=1G@0x80000000 text earlyprintk=ttyO2
initrd panda/uInitrd

PXE Booting

With the proper X-Loader and U-Boot files in place (at your first SD card partition), and with the TFTP + DHCP server also properly installed, you can just jump and try TFTP/PXE boot.

Stop the U-Boot autoload and call the following commands:

setenv pxecfg_ram 0x88000000: location in RAM to load the pxecfg file
setenv kernel_ram 0x80000000: location in RAM to load the kernel
setenv initrd_ram 0x81600000: location in RAM to load the initrd
setenv autoload no: disable autoload while calling bootp (so you can just set up your network without autoboot)
usb start: start USB and enables the SMSC95xx ethernet interface
bootp: initialize the network, probing the ip address settings from your DHCP server
pxecfg get: probe the pxecfg config file
pxecfg boot: boot :-)

You should get a similar output as:

Texas Instruments X-Loader 1.5.0 (Jul 11 2011 – 07:52:49)
Reading boot sector
Loading u-boot.bin from mmc

U-Boot 2011.06 (Jul 11 2011 – 02:49:51)

CPU : OMAP4430
Board: OMAP4 Panda
I2C: ready
DRAM: 1 GiB
MMC: OMAP SD/MMC: 0
Using default environment

In: serial
Out: serial
Err: serial
Net: No ethernet found.
Hit any key to stop autoboot: 0
Panda # setenv pxecfg_ram 0x88000000
Panda # setenv kernel_ram 0x80000000
Panda # setenv initrd_ram 0x81600000
Panda # setenv autoload no
Panda # usb start
(Re)start USB…
USB: Register 1313 NbrPorts 3
USB EHCI 1.00
scanning bus for devices… The request port(2) is not configured
EHCI timed out on TD – token=0x80008c80
The request port(2) is not configured
4 USB Device(s) found
scanning bus for storage devices… 0 Storage Device(s) found
scanning bus for ethernet devices… 1 Ethernet Device(s) found
Panda # bootp
Waiting for Ethernet connection… done.
BOOTP broadcast 1
DHCP client bound to address 10.42.43.10
Panda # pxecfg get
missing environment variable: pxeuuid
missing environment variable: ethaddr
Retreiving file: pxelinux.cfg/0A2A2B0A
Waiting for Ethernet connection… done.
Using sms0 device
TFTP from server 10.42.43.1; our IP address is 10.42.43.10
Filename ‘pxelinux.cfg/0A2A2B0A’.
Load address: 0x88000000
Loading: #
done
Bytes transferred = 239 (ef hex)
Config file found
Panda # pxecfg boot
Hit any key to stop autoboot: 0
Label: panda-natty
kernel: panda/uImage
append: console=ttyO2,115200n8 root=/dev/mmcblk0p2 ro fixrtc vram=48M omapfb.vram=0:24M mem=1G@0x80000000 text earlyprintk=ttyO2
initrd: panda/uInitrd
Retreiving file: panda/uInitrd
Waiting for Ethernet connection… done.
Using sms0 device
TFTP from server 10.42.43.1; our IP address is 10.42.43.10
Filename ‘panda/uInitrd’.
Load address: 0x81600000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
############
done
Bytes transferred = 3982715 (3cc57b hex)
Retreiving file: panda/uImage
Waiting for Ethernet connection… done.
Using sms0 device
TFTP from server 10.42.43.1; our IP address is 10.42.43.10
Filename ‘panda/uImage’.
Load address: 0x80000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#########################
done
Bytes transferred = 4174480 (3fb290 hex)
## Booting kernel from Legacy Image at 80000000 …
Image Name: Ubuntu Kernel
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4174416 Bytes = 4 MiB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum … OK
## Loading init Ramdisk from Legacy Image at 81600000 …
Image Name: Ubuntu Initrd
Image Type: ARM Linux RAMDisk Image (uncompressed)
Data Size: 3982651 Bytes = 3.8 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum … OK
Loading Kernel Image … OK
OK

Starting kernel …

Uncompressing Linux… done, booting the kernel.
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
…

This should be enough for you to get your Pandaboard booting with PXE. You can also script these commands at your boot.scr file that U-Boot loads automatically from your SD card, so you don’t have to call them by hand every time you reboot your board.

In case it doesn’t work for you, just ping me (rsalveti) at #linaro on freenode :-)

Anselmo Lacerda S. de Melo: Qt Contributors’ Summit

2011-06-17T12:22:07+00:00

We are in Berlin, attending the Qt Contributor’s Summit. 265 people are discussing very important (and interesting) topics for Qt 5 and the open governance. You can follow what’s happening here using the hash code #qtcs on several social networks. And of course, you can check the wiki page of the event.

Personally, I am attending sessions related with QML, Qt-Components, DevNet and open governance.

And yesterday was Thursday, also known as the “Yellow Day”:

Ok, Ademar was not wearing yellow t-shirt. Well, imagine him as the goalkeeper of the team ;)

Aloisio Almeida: Managing multiple network profiles using upstart

Aloisio — 2011-06-07T03:26:16+00:00

One of the purposes of buying the Jetway mini-top HBJC600C99-352-BW was using it as wireless router. I installed Ubuntu 11.04 on it. As NetworkManager does not handle wireless interfaces in access point (AP) mode, I decided to design my own network management. The idea is simple: use the required daemons/application and create upstart jobs to handle the activation logic. These are the network profiles I want to support:

WIFI: The htpc will connect to any known AP available.
ADSL: The htpc will connect to Internet via ADSL and act as a wireless router.
3G: The htpc will connect to the Internet via 3G and act as a wireless router.

List of applications I need to use:

upstart;
wpa_supplicant;
dhclient;
brctl;
ifconfig;
hostapd;
dhcpd;
pppd;
iptables;
dnsmasq.

To install the needed packages, type the application name in the terminal. Ubuntu will help you with package’s name.

There are thousand ways to do that using upstart, so let me list some constraints I kept in mind when designing the system:

Customized solution. I believe that generic solutions must be used only when you really need them. Every time you make something more generic you end up adding more logic, creating more corner cases and spending more time with use cases that you will NEVER face. Example of assumptions: my built-in ethernet port is eth0, my wireless interface wlan0 is NOT hotplug.
Always have a maintenance channel available. Shit happens, all the time. So independent of the network profile I choose I want some way to get a secure shell connection without connect a keyboard (that I don’t have) to the htpc.
One upstart job per long lived process. Among the applications listed above, some have short lifetime, i.e. they set some configuration and/or perform some action and finish. Others keep running until some orderly finish action. I decided that the formers don’t need to be a job, they can be run as pre or post scripts of other jobs.

1st step: Get rid of Ubuntu networking jobs.

I will manage my network stuff using only upstart jobs, so I don’t need network-manager and ifupdown utilities. So lets disable them:

root$ echo manual >> /etc/init/networking.conf
root$ echo manual >> /etc/init/network-interfaces.conf
root$ echo manual >> /etc/init/network-manager.conf
root$ echo manual >> /etc/init/networking.conf
root$ echo manual >> /etc/init/bridge-network-interface.conf

Now, we need a job to raise the loopback interface. The following job is completely based on the Ubuntu’s network-interface job:

/etc/init/loopback.conf

start on net-device-added INTERFACE=lo
stop on net-device-removed INTERFACE=lo

pre-start script
    mkdir -p /var/run/network
    ifconfig lo 127.0.0.1 up
    initctl emit -n net-device-up IFACE=lo LOGICAL=lo
                                  ADDRFAM=inet METHOD=loopback
end script

post-stop exec initctl emit -n net-device-down IFACE=lo LOGICAL=lo
                                               ADDRFAM=inet METHOD=loopback

As upstart will run and take care of all daemons listed above, we must disable the sysinit scripts that start/stop them (that ones in /etc/init.d).

root$ update-rc.d -f hostapd remove
root$ update-rc.d -f isc-dhcp-server remove
root$ update-rc.d -f dnsmasq remove

2nd step: create mutually exclusive jobs to set the network profile.

/etc/init/net-profile-adsl.conf

stop on (starting net-profile-adsl
         or starting net-profile-wifi)
# the 'start on' below means "i'm the default profile"
start on (local-filesystems
          and net-device-added INTERFACE=wlan0
	  and net-device-added INTERFACE=eth0)

etc/init/net-profile-3g.conf

stop on (starting net-profile-3g
         or starting net-profile-wifi)

/etc/init/net-profile-wifi.conf

stop on (starting net-profile-3g
         or starting net-profile-adsl)

The main ideia of this arrangement is to change the profile using a single command. For instance, if somehow my adsl connection become unavailable I just need to connect my cell phone to the htpc and run the following command:

root$ start net-profile-3g

To reach this goal, all the following jobs will depend on this set of “fake” jobs as you can see in the next steps.

3rd step: set-up the bridge

First, the question. Why do I need a bridge? It’s because I have a printer connected to the htpc via ethernet. As I only have one built-in ethernet port I bought a usb-ethernet adapter (eth1). So, every time the htpc is running the 3g or adsl profile, I want wlan0 and eth1 in a bridge to route the connection to them. I can even say that everytime I plug-in the usb-ethernet adapter, I want dhcpd sending valid IP addresses to it (do you remember that I want a ssh access anytime?). The solution that I chose:

The bridge (br0) will be set up at startup time and dhcpd will be configured to manage this interface
Whenever the usb-ethernet is plugged in, its interface will be added to the bridge
Every time ADSL or 3G profiles are chosen, wlan0 will also be added to the bridge

/etc/init/bridge.conf

start on started loopback
stop on stopping loopback

pre-start script
    brctl addbr br0
    ifconfig br0 10.0.1.1 netmask 255.255.255.0 broadcast 10.0.1.255
end script

post-stop script
    ifconfig br0 down
    brctl delbr br0
end script

/etc/init/usb-ethernet.conf

start on net-device-added INTERFACE=eth1
stop on net-device-removed INTERFACE=eth1

pre-start script
    ifconfig eth1 up
    brctl addif br0 eth1
end script
post-stop exec brctl delif br0 eth1

/etc/init/dhcpd.conf

start on started bridge
stop on stopping bridge

exec dhcpd -f

Note that the configurations above do not show when the wlan0 interface is added to the bridge. It will be covered in the next steps. Do not forget to change /etc/default/isc-dhcp-server in order to make dhcpd manage the br0 interface. Also, you need a valid configuration in the /etc/dhcp/dhcpd.conf. I will show my config below:

ddns-update-style none;

option domain-name-servers 10.0.1.1;

default-lease-time 600;
max-lease-time 7200;

authoritative;

log-facility local7;

subnet 10.0.1.0 netmask 255.255.255.0 {
  range 10.0.1.10 10.0.1.254;
  option routers 10.0.1.1;
}

4th step: managing wlan0 when the WIFI profile is chosen

Basically I need wpa_supplicant and dhclient running on wlan0 interface. Also, I need to keep wlan0 out of the bridge. Dumping the config:

/etc/init/wpa_supplicant.conf

start on started net-profile-wifi
stop on stopping net-profile-wifi

exec wpa_supplicant -iwlan0 -Dnl80211 -c/etc/wpa_supplicant.conf

/etc/init/dhclient.conf

start on started wpa_supplicant
stop on stopping wpa_supplicant

exec dhclient -d wlan0

emits net-device-up
emits net-device-down

post-start exec initctl emit -n net-device-up IFACE=wlan0 LOGICAL=wlan0
                                              ADDRFAM=inet METHOD=dhcp
pre-stop script
    ifconfig wlan0 0.0.0.0
    initctl emit -n net-device-down IFACE=wlan0 LOGICAL=wlan0
                                    ADDRFAM=inet METHOD=dhcp
end script

Note that you need a valid /etc/wpa_supplicant.conf file in order to be able to connect to a known AP. I don’t think I need to paste my config here. If you have any doubts run ‘man wpa_supplicant.conf’.

5th step: managing wlan0 when either 3G or ADSL profiles are chosen

I only need to launch hostpad and to add wlan0 to the bridge (br0). The job configuration:

/etc/init/hostapd.conf

start on (started net-profile-3g
          or started net-profile-adsl)
stop on (stopping net-profile-3g
         or stopping net-profile-adsl)

exec hostapd /etc/hostapd.conf

post-start script
    while [ "`iw wlan0 info | grep type | cut -d ' ' -f 2`" != "AP" ]; do
        sleep 0.5
    done
    brctl addif br0 wlan0
end script
pre-stop exec brctl delif br0 wlan0

Note how wlan0 is added to the bridge. The current solution is a ugly hack that I’m planning to improve further… The problem is that a wireless interface in ‘managed mode’ cannot be added to a bridge. So when I move from WIFI profile (the wlan0 is in managed mode) to 3G or ADSL profile (wlan0 is in master mode), the brctl addif command fails. The reason is that hostapd daemon does not have enough time to set the master mode before its own post-start script runs.

Lets remind you again about valid configuration files… My wireless device uses ath9k kernel driver and below I’m pasting my hostapd.conf

/etc/hostapd.conf

interface=wlan0
bridge=br0
driver=nl80211

ssid=myssid
hw_mode=g
channel=1

wme_enabled=1
ieee80211n=1
ht_capab=[HT40+][SHORT-GI-40][TX-STBC][RX-STBC1][DSSS_CCK-40]

macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=come_on
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

6th step: managing ppp0 interface when either ADSL or 3G profiles are chosen

I only need to call pppd with the desired profile and create a NAT using iptables. There are bare differences between 3g and adsl pppd jobs as you can notice below:

/etc/init/pppd-gvt.conf

start on started net-profile-adsl
stop on stopping net-profile-adsl

emits net-device-up
emits net-device-down

pre-start exec ifconfig eth0 0.0.0.0 up

exec pppd call gvt

post-start script
    iptables -t nat -A POSTROUTING -s 10.0.1.0/24 -o ppp0 -j MASQUERADE
    iptables -A FORWARD -s 10.0.1.0/24 -o ppp0 -j ACCEPT
    iptables -A FORWARD -d 10.0.1.0/24 -m conntrack
                --cstate ESTABLISHED,RELATED -i ppp0 -j ACCEPT
    initctl emit -n net-device-up IFACE=ppp0 LOGICAL=ppp0
                                  ADDRFAM=inet METHOD=ppp
end script

pre-stop script
    iptables -F
    initctl emit -n net-device-down IFACE=ppp0 LOGICAL=ppp0
                                    ADDRFAM=inet METHOD=ppp
end script

pre-start exec ifconfig eth0 down

/etc/init/pppd-vivo3g.conf

start on started net-profile-3g
stop on stopping net-profile-3g

emits net-device-up
emits net-device-down

exec pppd call vivo3g

post-start script
    iptables -t nat -A POSTROUTING -s 10.0.1.0/24 -o ppp0 -j MASQUERADE
    iptables -A FORWARD -s 10.0.1.0/24 -o ppp0 -j ACCEPT
    iptables -A FORWARD -d 10.0.1.0/24 -m conntrack
                --cstate ESTABLISHED,RELATED -i ppp0 -j ACCEPT
    initctl emit -n net-device-up IFACE=ppp0 LOGICAL=ppp0
                                  ADDRFAM=inet METHOD=ppp
end script

pre-stop script
    iptables -F
    initctl emit -n net-device-down IFACE=ppp0 LOGICAL=ppp0
                                    ADDRFAM=inet METHOD=ppp
end script

Important notes:

My adsl modem is in bridge mode and connected to eth0 port.
GVT is my adsl provider name and Vivo3G is my 3G operator name

Configs to adsl connection:

/etc/ppp/peers/gvt

noipdefault
defaultroute
hide-password
noauth
usepeerdns
user "turbonet@turbonet"
plugin rp-pppoe.so eth0
remotename gvt
ipparam gvt

Add the following line to /etc/ppp/pap-secrets

"turbonet@turbonet" gvt "gvt25"

Configs to 3G connection:

/etc/ppp/peers/vivo3g

hide-password
noauth
connect "/usr/sbin/chat -v -f /etc/chatscripts/vivo3g"
/dev/ttyACM0
115200
defaultroute
noipdefault
user "vivo"
remotename vivo3g
ipparam vivo3g
usepeerdns

/etc/chatscripts/vivo3g

ABORT BUSY ABORT 'NO CARRIER' ABORT VOICE ABORT 'NO DIALTONE' ABORT 'NO DIAL TONE' ABORT 'NO ANSWER' ABORT DELAYED
'' ATZ
OK AT+CGDCONT=1,"IP","zap.vivo.com.br"
OK "ATDT*99#"
CONNECT ''

Add the following line to /etc/ppp/pap-secrets

"vivo" vivo3g "vivo"

7th (and last) step: Setup local dns server

To configure the htpc to run as a name server, I’m using dnsmasq:

/etc/init/dnsmasq.conf

start on started loopback
stop on stopping loopback

exec dnsmasq -k -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new

/etc/dnsmasq.conf

interface=br0
no-dhcp-interface=br0

If you want to use dnsmasq locally, add this line to your /etc/dhcp/dhclient.conf

prepend domain-name-servers 127.0.0.1;

That’s all. This setup is not fully tested (stressed). If you read/applied this and have some suggestions, please reply here

Aloisio Almeida: New gadget!

Aloisio — 2011-05-31T01:44:19+00:00

I was looking for a computer with following features:

Act as a wireless router
Act as a multimedia center (1080p)
Quiet
Low power consumption

After some research, I chose the Jetway Mini-Top HBJC600C99-352W-BW.

Software solution: Ubuntu 11.04 + XBMC. The installation has worked out of the box, no main issues. XBMC standalone is launched by default. Nice job xmbc team!!! The software is awesome.

In the following posts I will show how I customized Ubuntu to make it fit my requirements

Anselmo Lacerda S. de Melo: Meego Conf 2011 – San Francisco

2011-05-23T13:00:25+00:00

I am in San Francisco (again! =), now for the MeeGo Conference 2011.

The event is about to begin with the keynote “The Future of MeeGo Starts Now” presented by the president of Linux Foundation, Jim Zemlin.

This time, I’m going to present a talk in the event too! The talk is today and the topic is “Writing applications for multiple MeeGo devices”.

There are other talks by openBossa/INdT guys, check the schedule!

Ian Lawrence: Evaluating projects for their return on social investment

2011-04-25T12:23:01+00:00

I have just published the software I wrote for my MBA in Strategic Management of Technology Innovation.

A slightly customized version of the software is currently in use as a innovation funnel at INdT.

It is a commonly accepted fact in business that social projects bring some tangible beneﬁts to an organization. Unfortunately, even if the project is very successful this often does not impact directly on the ﬁnancial bottom line. And, in a world of razor-thin margins, a set of activities that drive up corporate cost without any directly identiﬁable return is a tough sell, no matter how worthwhile and noble the project might be. The business case for social projects is therefore contingent on ﬁnding a suitable method for valuation — one that allows managers to understand the implications of an indirect beneﬁt and then make ’intelligent’ decisions about which projects to choose and the most feasible level of resources to commit.

I approached the problem using techniques from agile development. Scrum sizings were run with stakeholders experienced in the type of project under evaluation which estimated the intangible values present in the social project.The values were then passed into an algorithm which ran an economic analysis of the project with its associated series of estimates as projected 'cash-flows'. The economic indicators calculated are net present value, internal rate of return and eﬃciency of investment

Projects are then ranked on their efficiency of social investment score.

The customization used at INdT is that instead of using scrum estimates the software has values assigned by answering questions. This makes the software then useful for all kinds of projects.The following questions were used

What is the quality of information presented.?
How much does this project align with strategy?
What is the probability that the project will build competencies?
The project opens new, significant markets for customers (What is the potential for market and revenue creation?)
What is the project's capacity to leverage competitive position (Impact on competition)
If the project is approved what is the span of possible applications that the resulting product might be applied to?
Are there any IPR implications?
What is the technological feasibility of the project?
Assuming the project is implemented the set up costs of it are likely to be?
What are the technological risks?

and the software also has a 'stage-gate' built in now so that projects can be eliminated early on if some basic requirements are not met. Things like the project conflicting with the current portfolio or not adding value fit this description.

The software is published under the AGPL and it uses a standard Django set up. It should be relatively bug free but YMMV depending on what you want to do with it. The full thesis - How to choose social investment projects by calculating their eﬃciency of investment using agile software development techniques - is available to download here. Enjoy!

Renato Oliveira: Google Summer of Code: PySide under PSF umbrella

Renato Araujo — 2011-03-29T16:49:04+00:00

PySide was accepted as a project under PSF umbrella. Any student who want to submit ideas related with PySide can contact us, or any PySide mentor, to discuss the ideas and prepare some proposals. Feel free to propose new ideas, do not get stuck only in the current ones be creative and good luck. You can get more information on Python GSoC wiki, PySide GSoC wiki, and #pyside on freenode.

Ian Lawrence: Software shortlisted for sustainability award

2011-03-11T09:32:11+00:00

NDG has been shortlisted for the Just Means 2011 – Social Innovation Awards, an important program which recognizes innovative software in the social and environmental areas. The result will be published during the event “2011 Redefining Value”, which will occur on the 25th of March, in London.

The software is competing in the category ‘Most Strategic Use of Philanthropic Funds’, disputing the prize with other three companies – Nature’s Path Foods, Oracle Corporation and Barclays PLC.

“The initiative to submit NDG to the prize came from Nokia's Corporate Social Investment team. Those involved believe that the work done recently with the software has brought many positive effects to communities in need worldwide and for this reason we have the potential to compete,” explains Ian Lawrence, a software developer at INdT, whom along with Rodrigo Ramos – also an INdT developer – works on the NDG project in Manaus, developing new features and offering support to NGOs and companies which wish to use the tool.

NDG was launched in 2008, and it is used for the collection of field data, substituting the use of paper survey forms and allowing the secure transfer of this information to a server, in real time. In 2009, the State of Amazonas’ Secretary of Health (SUSAM) adopted the software to conduct a campaign to fight Dengue Fever in Amazonas.

The results were so significant that NDG was adopted worldwide by many different organizations. Examples are the Philippines’ Government who along with the WWF use the tool to map the agricultural sector in the country and the Kenyan Government who run child birth registration programs in rural areas.

Marcelo Lira: Python bindings para libepub usando Shiboken

2011-03-09T03:10:12+00:00

Tenho um Kindle, gosto muito dele, mas infelizmente não posso dizer o mesmo sobre o formato usado para os livros. Não que eu tenha feito um estudo detalhado do formato mobi e chegado a conclusão que ele é tecnicamente inferior. Não. O problema é que ePub é tão mais popular, o que significa mais livros, e mais ferramentas com as quais brincar. Calibre é ótimo para converter formatos de eBooks, mas eu prefiro a simplicidade de baixar um livro, copiar pro dispositivo e já sair lendo.

Quando meu Kindle não está à mão, meu leitor substituto é um N900 com MeBook, um dos meus aplicativos preferidos – ele procura por livros (no formato ePub) no Feedbooks, baixa pro dispositivo e os mostra numa lista bem bacana, com capas e tudo. Examinando o código fonte do MeBook aprendi que ele usa a libepub para ler eBooks ePub, então pensei que seria interessante fazer bindings Python para libepub. E aqui estamos.

Ah, e mais uma coisa antes de começarmos, encontrei um projeto Python no github chamado libepub. Não verifiquei ainda, mas o menciono aqui pelo bem da informação.

Shiboken

Shiboken é a ferramenta de renome mundial usada para gerar o mundialmente famoso PySide (que ocore ter alcançado a versão 1.0 semana passada – viva!) Ela é uma das melhores ferramentas disponíveis para gerar bindings Python para bibliotecas C++. Mas isso sou eu falando – eu nela. Contudo, libepub é escrita em C, e mesmo que Shiboken possa gerar bindings para um punhado de funções globais e juntá-las num módulo Python, fica uma porcaria. De forma que terei de fazer algumas preparações para que a libepub apareça bela em Python.

Pré-requisitos: arquivos de desenvolvimento (i.e. os cabeçalhos) para ApiExtractor, GeneratorRunner, Shiboken, e libepub. (Para usuários de Debian/Ubuntu isto significa: libapiextractor-dev, libgenrunner-dev, libshiboken-dev, and libepub-dev.) E o compilador C++ mais o CMake, é claro.

Lembrete: ApiExtractor, GeneratorRunner e Shiboken são feitos com Qt (somente módulos principais, nenhuma delas tem ui), mas os bindings gerados não dependem da Qt de forma alguma. Exceto, obviamente, se a biblioteca para qual os bindings serão gerados já a use.

libepub

Primeiro, vamos dar uma olhada na libepub. Ela tem três estruturas usadas como ponteiros opacos:

// Contem informacao sobre o arquivo ePub.
struct epub;
// Objeto iterador para o indice.
struct titerator;
// Objeto iterador para o conteudo do livro.
struct eiterator;

Se você quiser ler o conteúdo de um arquivo ePub, chame a função que irá criar uma estrutura epub, itere por seu índice com um titerator, e depois pelos conteúdos mesmos usando um eiterator.

A maioria das funções seguem aquele formato “C orientado a objetos” visto em outras bibliotecas, com o primeiro argumento sendo um ponteiro para a estrutura que representa o “this” ou “self” em linguagens OO. Exemplos:

void
epub_dump(struct epub* epub);

unsigned char**
epub_get_metadata(struct epub* epub,
enum epub_metadata type,
int* size);

int
epub_get_data(struct epub* epub,
const char* name,
char** data);

CMake

Usarei CMake como o sistema de compilação dos bindings pois me sinto confortável com ele – é o mesmo usado no PySide e no conjunto de ferramentas de geração do Shiboken. Não darei muita atenção à esta parte do processo, apenas veja os arquivos CMakeLists.txt; explicação mais detalhada sobre o processo de compilar um binding pode ser encontrada em PySide Binding Generation Tutorial. Para informação realmente básica, veja PySide CMake Primer.

C++ Wrapper feito à mão para uma biblioteca em C

Para gerar bindings Python para uma biblioteca C++, o sujeito precisa escrever uma descrição XML chamada type system, que declarará o quê precisa ser exposto para Python, e o quê sofrerá modificação: funções globais, classes, namespaces, enums. Por exemplo, se em C++ temos a classe Rectangle, eu a declararia no type system dessa forma:

<value-type name='Rectangle' />

Todos os métodos pertencentes à Rectangle serão expostos em Python automaticamente. O mesmo não vale para as funções em C que representam os métodos para a estrutura epub. Infelizmente não há modo de dizer para o gerador Shiboken que quero a estrutura epub e suas funções representadas como uma classe com métodos, sendo assim precisamos escrever um fino envoltório C++ ao redor das estruturas C. Isto seria um bocado de trabalho para uma grande biblioteca C, mas mesmo com uma minúscula sinto o desconforto de precisar recorrer a este tipo de gambiarra. O type system deveria ser expressivo o bastante para gerar bindings para estruturas e funções em C como se elas fossem objetos e métodos apropriados. Vou marcar isto para futuras melhorias comunitárias/fora-do-trabalho no Shiboken.

Vejamos um pedaço de código do arquivoepub_cpp_wrapper.h:

class EPub {
public:
...

~EPub() { epub_close(m_epub); }

static inline EPub*
open(const char* filename, int debug = 0) {
struct epub* book = epub_open(filename, debug);
if (book)
return new EPub(book);
return 0;
}

...
inline void dump() { epub_dump(m_epub); }

inline int
get_data(const char* name, char** data) {
return epub_get_data(m_epub, name, data);
}

...

private:
explicit EPub(struct epub* ptr) : m_epub(ptr) {}
EPub(const EPub& other) {}
EPub& operator=(const EPub& other) {}
struct epub* m_epub;
};

Note que todos os métodos são marcados como inline para fazer este envoltório o mais fino possível. (GCC, estou olhando pra você!)

Epub::open

A função epub_open se torna o método estático EPub::open que retornará um novo objeto EPub para o arquivo ePub dado pelo parâmetro filename, ou um null pointer se o arquivo for inválido ou não existir.
O construtor para esta classe foi feito privado de forma que o único modo de criar objetos EPub é via EPub::open, que nunca criará um objeto EPub inválido.

~Epub

Em C o responsável por liberar a estrutura epub é epub_close, mas não criarei um método EPub::close(), pois o equivalente C++ para ele é o destrutor da classe.

Um binding Python gerado para o que temos até agora seria mais ou menos assim:

import epub
book = epub.EPub.open('sample.epub')
title = book.get_metadata(epub.EPUB_TITLE)

Claro que não expliquei como isto seria gerado, que o módulo se chamaria epub, o que é epub.EPUB_TITLE, e como Python saberia o que fazer com unsigned char**, mas tenhamos paciência.

Movendo enums por aí

EPUB_TITLE é um valor vindo do enum epub_metadata, se ambos forem exportados para Python conforme estão, eles terão esta aparência:

import epub
epub.epub_metadata
epub.EPUB_TITLE

Assim fica bem feio e fuleiro. Já que epub_metadata é um enum relacionado ao objeto epub (como nos informa o prefixo epub_), será natural que ele seja movido para dentro da class EPub. No meu mundo de fantasia, a tag do type system que descreve um enum C++ para Python teria a opção de movê-lo para dentro de outro objeto, e também de ser renomeado. Algo desse tipo:

<enum-type name='epub_metadata'
rename='metatada'
move-into='EPub'
remove-enum-value-prefix='EPUB_' />

E aqui temos outro item para a lista de “por fazer” do Shiboken. Enquanto esta funcionalidade não é implementada, terei de fazê-lo manualmente.

class EPub {
public:
enum metadata {
ID = int(EPUB_ID),
TITLE = int(EPUB_TITLE),
CREATOR = int(EPUB_CREATOR),
CONTRIB = int(EPUB_CONTRIB),
SUBJECT = int(EPUB_SUBJECT),
PUBLISHER = int(EPUB_PUBLISHER),
DESCRIPTION = int(EPUB_DESCRIPTION),
DATE = int(EPUB_DATE),
TYPE = int(EPUB_TYPE),
FORMAT = int(EPUB_FORMAT),
SOURCE = int(EPUB_SOURCE),
LANG = int(EPUB_LANG),
RELATION = int(EPUB_RELATION),
COVERAGE = int(EPUB_COVERAGE),
RIGHTS = int(EPUB_RIGHTS),
META = int(EPUB_META)
};
...
inline unsigned char**
get_metadata(metadata type, int* size) {
return epub_get_metadata(m_epub,
epub_metadata(type),
size);
}
...
};

Os valores do enum epub_metadata foram convertidos para int para impedir o ApiExtractor de emitir toda sorte de warnings dizendo que ele não sabe quem são esses caras.

De qualquer modo, é terrível… ter esse gerador incrementado, e ter de escrever tudo isso… nãããão!

TIterator e EIterator

As estruturas C titerator e eiterator serão envolvidas pelas class C++ TIterator e EIterator, respectivamente, e como a classe EPub seus construtores serão privados.

class TIterator {
public:
enum type {
NAVMAP = int(TITERATOR_NAVMAP),
GUIDE = int(TITERATOR_GUIDE),
PAGES = int(TITERATOR_PAGES)
};
~TIterator() { epub_free_titerator(m_iter); }
inline bool isValid() {
return epub_tit_curr_valid(m_iter);
}
...
private:
friend class EPub;
explicit TIterator(struct titerator* iter)
: m_iter(iter) {}
struct titerator* m_iter;
};

Novas instâncias de TIterator e EIterator são criadas por métodos de EPub, por isso ela deve ser feita amiga (friend) das classes iteradoras.

class EPub {
public:
...
inline EIterator*
get_iterator(EIterator::type type, int opt = 0) {
struct eiterator* it = epub_get_iterator(m_epub, eiterator_type(type), opt);
if (it)
return new EIterator(it);
return 0;
}
inline TIterator*
get_titerator(TIterator::type type, int opt = 0) {
struct titerator* it = epub_get_titerator(m_epub, titerator_type(type), opt);
if (it)
return new TIterator(it);
return 0;
}
...
};

python-epub

Agora é hora de dizer ao gerador o quê entra e o quê será modificado nos bindings.

O cabeçalho global

O cabeçalho global é um arquivo que inclui todos os outros cabeçalhos da biblioteca que será analizada. No cabeçalho global o desenvolvedor de bindings pode também adicionar algumas cláusulas, como um #define que irá causar alguma condição de mudança nos cabeçalhos da biblioteca alvo, que irá afetar o binding gerado.

O arquivo epub_global.h que uso aqui é muito simples:

#ifndef EPUB_GLOBAL_H
#define EPUB_GLOBAL_H

#include <epub.h>
#include <epub_shared.h>
#include <epub_version.h>

#include "epub_cpp_wrappers.h"

// ApiExtractor reclamara' se
// encontrar apenas pre-definicoes.
struct titerator {};
struct eiterator {};

#endif

Como o comentário nos diz, ApiExtractor não gosta quando encontra pré-declarações sem definições. Escolhi adicionar estas definições falsas para struct titerator e struct eiterator. (Por alguma razão o gerador nada diz sobre struct epub.) Outra opção seria não usar estas definições falsas, mas adicionar uma linha ao arquivo type system dizendo ao gerador para ignorar avisos relativos à estas estruturas.

A descrição do Type System

Aqui segue um pedaço de código do arquivo typesystem_epub.xml.

<?xml version='1.0'?>
<typesystem package='epub'>
...
<rejection enum-name='epub_metadata' />
...
<object-type name='EPub'>
<enum-type name='metadata' />
...
</object-type>
</typesystem>

O enum epub_metadata não é exportado para Python (ao menos não diretamente) e para evitar que o gerador emita seus alertas, ele precisa ser explicitamente rejeitado; seu substituto C++ é adicionado depois dentro do object-type EPub. O mesmo acontece com os outros enums.

Na notação do type system object-type se refere aos objetos que são passados somente como ponteiros (como EPub, cujos construtor e operador de cópia são privados). Se, de outro modo, o objeto pode ser passado como valor, ele deve ser declarado como value-type, conforme nosso exemplo de Rectangle mencionado anteriormente.

O Protocolo de Iterador de Python

Em Python, se um objeto suporta o Protocolo de Iterador, poderei usar em comandos for, desta forma:

from epub import EPub, TIterator
book = EPub.open(self.epub_file)
for toc_it in book.get_titerator(TIterator.NAVMAP):
if not toc_it.isValid():
continue
print 'link : ' + toc_it.link()
print 'label: ' + toc_it.label()

Seguir o Protocolo de Iterador de Python consiste tão somente de um objeto implementar os métodos __iter__() e __next__().

Nota: o nome correto para o método é next() e não __next__(). Este é apenas um bug menor no gerador, que acabei de encontrar.

O XML para adicionar as funcionalidades do protocolo de iterador à classe TIterator será extenso, de forma que o partirei em duas partes, a primeira lidando com templates de código do type system.

Os templates do Type System

<?xml version='1.0'?>
<typesystem package='epub'>
...
<template name='iterator.__iter__'>
Py_INCREF(%PYSELF);
%PYARG_0 = %PYSELF;
</template>
<template name='iterator.__next__'>
if (%CPPSELF.next()) {
<insert-template name='iterator.__iter__' />
} else {
PyErr_SetNone(PyExc_StopIteration);
}
</template>
...

Os métodos de iterador para TIterator e EIterator terão exatamente a mesma implementação, então é esperto usar os templates do type system, e ter o código, e seus eventuais bugs, num único lugar.

O iterator.__iter__ método precisa apenas returnar o próprio objeto com seu contador de referências incrementado em um. iterator.__next__ chama o next() do objeto C++ subjacente (e também retorna a si próprio, e incrementa o refcounter), e sobe uma exceção Python StopIteration quando chega ao fim.

As variáveis do type system %PYSELF, %PYARG_0 e %CPPSELF são substituídas por valores dependentes do contexto onde são usados (e.g. as classes TIterator ou EIteration). Verifique a documentation para saber seus significados.

Adicionando os métodos de iteração

...
<object-type name='TIterator'>
...
<modify-function signature='next()' remove='all' />
<add-function signature='__iter__' return-type='PyObject*'>
<inject-code class='target' position='beginning'>
<insert-template name='iterator.__iter__' />
</inject-code>
</add-function>
<add-function signature='__next__' return-type='PyObject*'>
<inject-code class='target' position='beginning'>
<insert-template name='iterator.__next__' />
</inject-code>
</add-function>
</object-type>
...
</typesystem>

Primeiro o next() original do C++ é removido, então aqueles para o protocolo de iterador do Python são adicionados, usando a tag <insert-template/> para inserir o código personalizado definido anteriormente. A exatas mesmas linhas serão adicionadas à classe EIterator.

Só mais um pouco de gambiarras

Apenas um obstáculo permanece no caminho de ter iteradores Python apropriados. Quando o comando for de Python for usado para iterar sobre um objeto iterável, na primeira rodada ele chama o método __iter__ do objeto, e imediatamente após ele chama next, e continua chamando next para cada nova iteração.

O problema aqui é que o iterador C subjacente retorna um objeto levando conteúdo apropriado quando __iter__ é chamado, então a forma como a iteração do for Python funciona fará o primeiro item válido ser pulado. Uma gambiarra para este caso é usar uma flag no wrapper C++ que verificará se o iterador acabou de ser criado, de forma que ele não pulará para o iterador seguinte quando next for chamada pela primeira vez.

class TIterator {
public:
...
inline bool next() {
if (m_isFirst) {
m_isFirst = false;
return true;
}
return epub_tit_next(m_iter);
}
private:
friend class EPub;
explicit TIterator(struct titerator* iter)
: m_iter(iter), m_isFirst(true) {}
struct titerator* m_iter;
bool m_isFirst;
};

Pude ter esta liberdade porque TIterator é uma classe completamente sob meu (eu, o desenvolvedor do binding) controle. Se struct titerator fosse uma classe C++ desde o começo, esta solução não seria a melhor. Talvez se a libshiboken (a biblioteca de apoio usada por todos os bindings gerados pelo Shiboken) fornecesse uma classe iteradora base para lidar com essa diferença particular entre iteradores Python e C++. Ou talvez as classes geradas, quando identificadas como iteráveis pela presença dos métodos do protocolo de iteração adicionados pelo desenvolvedor do binding, poderiam ter tal funcionalidade. A última opção me parece a melhor, e esse é mais um item pra minha lista de melhorias futuras.

Conversões Personalizadas

Retornando valores unicode

Os métodos EIterator::curr() e EIterator::curr_url() returnam valores do tipo char*, que não tem um conversor (const char* é que tem), por isso escrevi um código específico para converter isto para unicode do Python.

<?xml version='1.0'?>
<typesystem package='epub'>
...
<template name='return_char_pointer'>
char* %0 = %CPPSELF.%FUNCTION_NAME();
if (%0) {
%PYARG_0 = PyUnicode_DecodeUTF8(%0, strlen(%0), "strict");
} else {
Py_INCREF(Py_None);
%PYARG_0 = Py_None;
}
</template>
...
<object-type name='EIterator'>
...
<modify-function signature='curr()'>
<inject-code class='target' position='beginning'>
<insert-template name='return_char_pointer' />
</inject-code>
</modify-function>
<modify-function signature='curr_url()'>
<inject-code class='target' position='beginning'>
<insert-template name='return_char_pointer' />
</inject-code>
</modify-function>
...
</object-type>
...
</typesystem>

Modificando a assinatura de um método

As assinaturas de alguns métodos C++ não podem ser automaticamente convertidas para código Python que faça sentido, então adicionei mais um pouco de código feito sob medida para tratar das situações caso a caso. Por exemplo, o método

unsigned char**
EPub::get_metadata(metadata type, int* size);

O argumento int* size recebe um ponteiro para um int, que conterá o tamanho da lista de strings unicode retornada como unsigned char**. Em Python ele meramente retornará uma lista de objetos unicode, e a chamada à get_metadata não precisa do argumento size.

Creio que a descrição do type system será o bastante para ver como o sistema de modificação funciona.

Baixando, compilando, etc.

Chega de explicações, agora vamos experimentar o código. Lembre-se que precisará também dos arquivos de desenvolvimento que mencionei muito tempo atrás

Clone a versão mais recente do repositório git:

git clone git://github.com/setanta/python-epub.git

ou baixe o tarball: python-epub.tar.bz2

No diretório do código fonte crie um diretório build e…

cd python-epub
mkdir build
cd build
cmake ..
make
ctest

O último comando merece uma conversa à respeito.

Testar, Testar, Testar

Quando trabalhando com desenvolvimento de bindings há uma miríade de coisas que podem dar errado, e muitas delas dão errado em completo silêncio. Tendo isto em mente, eu digo-lhe que ter testes de unidade torna a vida do desenvolvedor de bindings suportável.

Veja o diretório de testes para exemplos de como usar o bingind python-epub.

Para resultados detalhados dos testes, rode ctest com a opção -V (verboso).

Exemplo de UI (senão seria tedioso)

Mas a própria UI é bem tediosa, embora feita com o espetacular PySide, ela nem sequer mostra imagens. Por outro lado, com um pouco de código agora posso ver o conteúdo de um ebook ePub.

De dentro do diretório python-epub/build chame o visualizador de ePub dessa forma:

python ../simple-ui/bookviewer.py ../tests/beyond-the-wall-of-sleep.epub

Ele sempre espera um parâmetro com o caminho para um arquivo ePub, neste caso “Beyond the Wall of Sleep” de Lovecraft.

Screenshots:

Marcelo Lira: Python bindings for libepub using Shiboken

2011-03-09T00:31:54+00:00

I have a Kindle, I like it very much, but unfortunately I can’t say the same about the format it used for books. Not that I have made a detailed study of the mobi format and came to the conclusion that it is technically inferior. No. The problem is that ePub is so much more popular, which mean more books, and more tools to play with. Calibre is awesome to convert between eBook formats, but I prefer the simplicity of downloading a book, copying it to the device and reading right away.

When my Kindle is not at hand, my replacement reader is a N900 with MeBook, one of my favorite apps – it looks for books (in ePub format) on Feedbooks, downloads to device and show them in a nice list with covers and all. Looking at MeBook source code I learned that it uses libepub to read the ePub eBooks, so I thought it would be interesting to make a Python binding to libepub. And here we are.

Ah, and one more thing before we start, I found a related Python project on github called libepub. Hadn’t checked it yet, but I mention it here for information’s sake.

Shiboken

Shiboken is the world famous tool used to generate the world famous PySide bindings (that happen to have reached 1.0 last week – yay!) It is one of the best tools around to generate Python bindings for C++ libraries. But that’s me talking – I work on it. But libepub is written in C, and even though Shiboken can generate bindings for a bunch of global functions and put them together in a Python module, that’s crappy. So I’ll have to make some preparations to make libepub appear beautiful in Python.

Pre-requisites: development files (i.e. the headers) for ApiExtractor, GeneratorRunner, Shiboken, and libepub. (For Debian/Ubuntu users this means: libapiextractor-dev, libgenrunner-dev, libshiboken-dev, and libepub-dev.) And the C++ compiler plus CMake, of course.

Remainder: ApiExtractor, GeneratorRunner and Shiboken are made with Qt (only the core libs, no ui for them), but the bindings generated do not depend on Qt at all. Except, obviously, if the library being wrapped already does.

libepub

First let’s have an overview of libepub. It has three structures used as opaque pointers:

// Contains information about the ePub file.
struct epub;
// Iterator object for the Table of Contents.
struct titerator;
// Iterator object for the book contents.
struct eiterator;

If you want to read the contents of an ePub file, call the function that will create an epub structure, iterate through its table of contents with a titerator, and then the contents themselves with an eiterator.

Most functions follow that “object oriented C” format seen in other libraries, with the first argument being a pointer to the structure that represents the “this” or “self” in OO languages. Examples:

void
epub_dump(struct epub* epub);

unsigned char**
epub_get_metadata(struct epub* epub,
enum epub_metadata type,
int* size);

int
epub_get_data(struct epub* epub,
const char* name,
char** data);

CMake

I’ll use CMake as the build system for the bindings because I feel comfortable with it – it is the same used in PySide and all the Shiboken generator tool chain. I’ll not give too much attention to this part of the process, just check the CMakeLists.txt files; more detailed explanation on the process of building a binding can be found in the PySide Binding Generation Tutorial. For really really basic information check PySide CMake Primer.

Handmade C++ Wrapper for a C library

To generate Python bindings for a C++ library one must write a XML description called type system, which will declare what must be exposed on Python land, and if any of this needs modification: global functions, classes, namespaces, enums. For example, if in C++ I have the class Rectangle, I would declare it in the type system this way:

<value-type name='Rectangle' />

All the methods belonging to Rectangle would be exposed to Python automatically. The same is not true for the C functions that representing the methods for the epub struct. Unfortunately there’s no way to tell the Shiboken generator that I want to have the epub structure and functions represented as a class with methods, so we have to make a thin C++ wrapper around the C structures. That would be a lot of work for a huge C library, but even with a tiny one I feel uncomfortable having to resort for this kind of hackery. The type system should be expressive enough to bind C structs and functions as if they were proper objects with methods. I’ll mark this for future community/out-of-work improvements on Shiboken.

Let’s see a snippet from epub_cpp_wrapper.h:

Notice that all methods were marked as inline to make this wrapper as thinner as possible. (GCC, I’m looking at you!)

Epub::open

The epub_open function becomes the static method EPub::open that will return a new EPub object for the ePub file given by filename parameter, or a null pointer if the file is invalid or doesn’t exist.
The constructor for this class was made private so the only way to create EPub objects is via EPub::open, that’ll never create an invalid EPub.

~Epub

In C the responsible for freeing the epub structure is epub_close, but I’ll not make it EPub::close() because the C++ equivalent for it is the class’ destructor.

A generated Python binding for what we have until now would look roughly like this:

import epub
book = epub.EPub.open('sample.epub')
title = book.get_metadata(epub.EPUB_TITLE)

Of course I didn’t explained how this would be generated, that the module would be called epub, what is epub.EPUB_TITLE, and how Python would know what to do with unsigned char**, but bear with me.

Moving enums around

EPUB_TITLE is a value from the epub_metadata enum, if exposed to Python as they are, they’ll look like this:

import epub
epub.epub_metadata
epub.EPUB_TITLE

Which is pretty ugly and lame. Since epub_metadata is an enum related to the epub object (as the epub_ prefix tells us), it would be natural that it was moved inside the EPub class. In my fantasy world, the type system tag that describes a C++ enum to Python would have the option to move it inside another object, and also to be renamed. Something along these lines:

<enum-type name='epub_metadata'
rename='metatada'
move-into='EPub'
remove-enum-value-prefix='EPUB_' />

And here I have another thing for a Shiboken TODO list. While this feature is not implemented, I’ll have to do it manually.

The epub_metadata enum values were cast to int to prevent the ApiExtractor to emitting a bunch of warnings saying that it cannot tell who these guys are.

Anyways, it’s so awful… to have a fancy generator, and having to write all this… noooo!

TIterator and EIterator

The C structures titerator and eiterator will be wrapped by the C++ classes TIterator and EIterator, respectively, and like the EPub class their constructors are private.

New instances of TIterator and EIterator are created by EPub methods, because of that it must be a friend of the iterator classes.

python-epub

Now it’s time to say to the generator which goes in and which must change in the bindings.

The global header

The global header is a file that includes all other headers of the library that will be analyzed. In the global header the binding developer may also add some tweaks, like a #define that will trigger some condition in the target library headers, that will affect the generated binding.

The epub_global.h that I use here is very simple:

#ifndef EPUB_GLOBAL_H
#define EPUB_GLOBAL_H

#include <epub.h>
#include <epub_shared.h>
#include <epub_version.h>

#include "epub_cpp_wrappers.h"

// ApiExtractor complains if it finds only pre-definitions.
struct titerator {};
struct eiterator {};

#endif

As the commentary tells us, ApiExtractor doesn’t like when it finds forward declarations without definitions. I choose to add these two bogus definitions for struct titerator and struct eiterator. (For some reason the generator said nothing about struct epub.) Other option would be not to add those bogus definitions, but add a line to the type system file telling the generator to ignore warnings relative to those structures.

The Type System description

Here follows a snippet from typesystem_epub.xml.

<?xml version='1.0'?>
<typesystem package='epub'>
...
<rejection enum-name='epub_metadata' />
...
<object-type name='EPub'>
<enum-type name='metadata' />
...
</object-type>
</typesystem>

The epub_metadata enum is not exported to Python (at least not as it is) and to avoid the generator emitting his warnings, it must be explicitly rejected; its C++ substitute is added afterwards inside the EPub object type. The same happens to the other enums.

In the type system notation object-type refers to objects that are passed around solely as pointers (like EPub, whose copy constructor and operator are private). If, otherwise, the object can be passed as value, it should be declared as an value-type, as our Rectangle example mentioned before.

Python’s Iterator Protocol

In Python, if an object supports the Iterator Protocol I can use it on for statements, like this:

Following the Python Iterator Protocol consists solely of an object implementing the methods __iter__() and __next__().

Note: the correct name for the method is next() and not __next__(). That's a minor bug in the generator that I just found.

The XML to add iterator protocol features into TIterator class will be lengthy, so I'll split it into two parts, the first dealing with type system code templates.

Type System templates

The iterator methods for TIterator and EIterator have exactly the same implementation, so it’ll be smart to use type system templates and have the code, and its eventual bugs, in a single place.

iterator.__iter__ method just need to return the object itself with its reference counter incremented by one. iterator.__next__ calls the underlying C++ object’s next() (it also returns itself, and increments the refcounter), and raises a Python StopIteration exception when it reaches the end.

The type system variables %PYSELF, %PYARG_0 and %CPPSELF are replaced by values dependent on the context where they are used (e.g. TIterator or EIteration classes). Check the documentation for their meaning·

Adding the iterator methods

First the original C++ next() is removed, then the ones for the Python iterator protocol are added, using <insert-template/> tag to insert the previously defined custom code. Exactly the same lines will be added to EIterator class.

Just one more bit of hackery

Only one obstacle remains on the way of having proper Python iterators. When Python’s for statement is used to iterate through an iterable object, in the first round it calls the object’s __iter__ method, and immediately after it calls next, and keeps calling next for each new iteration.

The problem here is that our underlying C iterator returns an object loaded with proper content when __iter__ is called, then the way that Python’s for iteration works will cause the first item to be bypassed. A workaround for this case is to use a flag on the C++ wrapper that checks if the iterator has just been created, so that it will not move forward when next is called on it for the first time.

I had this freedom because TIterator is a class completely under my (me, the binding developer) control. If struct titerator were a C++ class from the beginning that approach would not be the best. Perhaps libshiboken (the supporting library used by all Shiboken generated bindings) should provide a base iterator class to handle this particular difference between Python and C++ iterators. Or perhaps the generated class, when identified as an iterable by the presence of iterator protocol methods added by the binding developer, should have such provisions. The latter options seems best, and that’s one more item for my list of future improvements.

Custom Conversions

Returning unicode values

The EIterator::curr() methods EIterator::curr_url() returns values of char* type, which doesn’t have a converter (const char* does have), so I’ve written a custom piece of code to convert it to Python’s unicode.

Modifying a method’s signature

Some C++ method signatures couldn’t automatically be converted to meaningful Python code, so I added more custom code to handle the situation on a case by case basis. For example, the method

unsigned char**
EPub::get_metadata(metadata type, int* size);

The int* size argument receives a pointer to an int, which will contain the size of the list of unicode strings returned as unsigned char**. In Python it would merely return a list of unicode objects, and the call to get_metadata doesn’t need the size argument.

I believe the type system description is enough to see how the modification system works.

Downloading, building, etc.

Enough with the explanations, now let’s try the code. Remember that you’ll need also development files that I mentioned long time ago.

Clone the latest version from the git repository:

git clone git://github.com/setanta/python-epub.git

or download the tarball: python-epub.tar.bz2

Inside the source code directory create a build directory and …

cd python-epub
mkdir build
cd build
cmake ..
make
ctest

The last command deserves some talking about.

Testing, Testing, Testing

When working with binding development there’s a myriad of things that can go wrong, and a number of them go wrong in complete silence. With this in mind, I tell you that having unit tests makes the binding developer life bearable.

To see detailed results from the tests, run ctest with -V (verbose) option.

Check the tests directory for examples on how to use the python-epub bindings.

UI Example (or else it would be boring)

But the UI itself is very boring, although it was made with the amazing PySide I did it as simple as possible, it doesn’t even show images. On the other hand, with very little code I can now see what’s inside an ePub ebook.

From python-epub/build directory call the ePub viewer like this:

python ../simple-ui/bookviewer.py ../tests/beyond-the-wall-of-sleep.epub

It always expects a parameter with the path to an ePub file, in this case “Beyond the Wall of Sleep” by Lovecraft.

Screenshots:

Ademar de Souza Reis Jr.: Férias 2011! Destino: Europa

2011-03-05T13:25:14+00:00

Depois de quase 2 anos sem férias e sem nenhuma viagem de “grande porte”, finalmente é chegada a hora de um merecido descanso do trabalho e de ir pra algum lugar ainda desconhecido, desbravar novas fronteiras.

Atualização: Álbum de fotos: Viagem à Europa 2011

O destino original que haviamos planejado ainda no ano passado era a Ásia, mais precisamente Índia e China com potencial de chegar até o Japão. Ir pra esses destinos por conta própria e com pouco tempo exige um consideravel esforço de planejamento e logística (vistos, idioma, grandes distâncias e altos custos em alguns lugares), então após alguns estudos, decidimos optar por um destino mais tranquilo e com melhor relação custo-benefício nesses aspectos: Europa.

Serão 35 dias no total (valeu Carnaval!), chegando por Paris e voltando por Milão. Com excessão das passagens de ida e volta, o roteiro está quase todo em aberto, pois como estamos em baixa temporada (final do inverno por lá), os riscos de problemas com hospedagem e transporte são mínimos.

O roteiro planejado, por enquanto, é esse:

Paris;
Londres;
Amsterdam e Rotterdam;
Berlin;
Moscow (ida e volta à Berlin);
Cidades diversas nos países do Leste Europeu (Polônia, República Checa, Eslovênia, Hungria?);
Viena;
Munique;
Zurique e região dos Alpes;
De Veneza até Roma;
Milão.

No mapa:

Visualizar Viagem Europa 2011 em um mapa maior

(suficiente pra cumprir o objetivo de 2 novos países em 2011) ;-)

Provavelmente não haverá tempo pra todos os destinos e o plano é alugar um carro em alguns desses trajetos, o que nos dará a oportunidade de conhecer cidades e vilas menores. Também existe a possibilidade de embarcarmos em voos baratos como os de EUR 7 da Ryan Air pra chegar em cidades mais remotas, ou mesmo mudar completamente a parte final da viagem.

Como sempre, será uma viagem sem exageros nos custos, com apoio de couchsurfers ao longo do caminho e muitas fotos. Pequenos relatos e fotos mais descompromissadas devem aparecer nos perfis do twitter @adebr e @tudocasa (Viviane) e em meu facebook.

Espero dessa vez pré-selecionar as fotos no caminho pra evitar o problema que tive com a viagem de 2009 à Patagônia, que não teve suas fotos publicadas até hoje (shame on me!). :-P

Em tempo, muito obrigado a todos os que nos ajudaram com dicas de roteiro e aos couchsurfers que nos ajudarão ainda mais durante o trajeto. :-)

Link Permanente | 2 commentários
Post tags: europa, viagem

Kenneth Christiansen: FOSDEM talk

Kenneth Rohde Christiansen — 2011-02-22T11:23:12+00:00

I will upload the slides soon.

[Update] Slides are up here at slideshare.net

Francisco Alecrim: Mandriva 2010 : rodando sua aplicação como administrador(root)

Francisco Keppler Silva Alecrim — 2011-02-21T18:16:37+00:00

Imagine que você desenvolveu uma aplicação que precisa de permissão de root. Agora você quer que o Mandriva chame sua aplicação como usuário administrador(root).

0 – Minha aplicação simples é um Hello world

echo -e '#include \nint main(){printf("hello mandriva world\\n");}' > hello.c
gcc hello.c -o hello

1 – Chamar o console-helper ao invés de chamar sua aplicação:

su - root
cd /usr/bin
ln -s /usb/bin/console-helper hello

2 – Criar arquivo PAM /etc/pam.d/hello

#%PAM-1.0
auth            include         config-util
account         include         config-util
session         include         config-util

3 – Criar arquivo com PATH em /etc/security/console.apps/hello

USER=root
PROGRAM=/home/alecrim/hello
SESSION=true
FALLBACK=true

Planet INdT

Ian Lawrence: Microsoft Education Delivery

Francisco Alecrim: SCRUM primeiros passos

Ian Lawrence: Jozian Ltd

Bruno Araujo: Unity3D: Typing effect with sound for Text UI components

Bruno Araujo: Unity3D: Using multiple Canvas components for switching UI components on the fly

Danilo Cesar: A brief discussion about Crosswalk’s JS, JAVA and Native Extensions performance

Adding the .so inside your apk

Performance measurement

Performance Results

Performance Results, round two

Conclusion

Links

Ademar de Souza Reis Jr.: Retrospectiva 2014

Bruno Araujo: Porfolio status: ONLINE

Ian Lawrence: Making Magic

South Sudan

What is MAGIC

Ian Lawrence: A Bitcoin project

Some words about hedging

What needs to be implemented?

What will be great to have in future?

What about security?

(rejected) Single bitcoin account managing all users accounts

(rejected) Separate bitcoin wallet for each user

Separate bitcoin wallet + cash-in-out wallet

How that will work?

What is the status of development?

Bruno Araujo: Tricky Python bugs: comparison madness

Anderson Lizardo: Controle de ponto usando dbus-monitor

Morpheuz: Open Academy and KDE

Everton Arruda: Get a list of TODOs and FIXMEs in your project

Danilo Cesar: YouCompleteMe: semantic auto-completion and JumpToDefinition for VIM on webkitgtk+

Prelude

YouCompleteMe

Description

Installation

vimrc and vundle

How it works

Tips&Tricks

Bruno Araujo: Unity: from zero to Pong

Bruno Araujo: short story: a single shot

Bruno Araujo: Developing games: the first steps

Bruno Araujo: Parsing a very large XML file with Python

Adenilson Cavalcanti: Considerations on home office

Ademar de Souza Reis Jr.: Retrospectiva 2012, planos pra 2013

Bruno Araujo: Welcome! Welcome to City 17…

Adenilson Cavalcanti: icecc setup how to

Ricardo Salveti: Pre-built images for XBMC (Ubuntu 12.04 based) with hw acceleration finally available at Linaro

Adriano Tinoco: QEdje

Adriano Tinoco: Hidra

Ricardo Salveti: Great start at Ubuntu Developer Summit (Q or 12.10) for ARM

Morpheuz: Big changes!

Ian Lawrence: Advances in mobile data collection: Nokia Data Gathering 4.04

Anselmo Lacerda S. de Melo: Qt 5 UiHelpers

Bruno Abinader: Moving out! :)

Anselmo Lacerda S. de Melo: Working in the open again

Ricardo Salveti: ARM Porting Jam this Friday!

Ricardo Salveti: Ubuntu TV UI at Pandaboard, and next steps

Ricardo Salveti: Ubuntu TV fully accelerated on a Pandaboard with Ubuntu LEB

Ricardo Salveti: HW video decode and XBMC support on a Pandaboard with Ubuntu LEB

Ademar de Souza Reis Jr.: Retrospectiva 2011, planos pra 2012

Ricardo Salveti: UDS-P/Linaro Connect Q4.11 and 11.11 cycle

Morpheuz: Back to life!

Danilo Cesar: Prague, Telepathy and HTML5 IM Client

A little bit about Telepathy Webkit Client

GObject introspection magic

Following a few screen-shots

Contact List

Chat Window

Receiving a call

The real thing running

Conclusions and future work

Ian Lawrence: Tangaza or 'announce' in Swahili

Morpheuz: It has been a while

Adenilson Cavalcanti: CellarDoor in German plus Debian package

Adenilson Cavalcanti: CellarDoor got its second translation: Italian!

Adenilson Cavalcanti: Presenting CellarDoor

Adenilson Cavalcanti: Long time no see you

Ricardo Salveti: launchpadlib: creating a changelog for a PPA