Many would have heard about Valgrind to detect memory leaks (among other functions it has), but unfortunately Valgrind does not work in Solaris SPARC, so we will have to use other tools when we work on a Solaris SPARC. In Oracle BRM documentation, Purify is mentioned, but since we have to pay a license, we are going to use something our OS already has so we don't have to pay for it.
To detect memory leaks (referred to * poid_t and * pin_flist_t) in the CM, the first thing we have to do is modify the CM's pin.conf adding this line:
- - disable_pcm_mempool 1
By adding that line in the CM's pin.conf we have disabled the memory pool that the CMs handle to process flist and poids, now the memory is allocated from the heap.
In the Oracle online documentation we can read: To reduce thread contention on malloc calls, CMs include a memory pool mechanism for processing flists and POIDs. When flists and POIDs are allocated memory from a pool, problems with memory leaks are hidden. To detect memory leaks in your CM, before you run Purify or any other diagnostic utilities to test memory usage, disable the memory pool by adding the following entry in the CM pin.conf file so that memory is allocated from the system heap.
(Remember that when we modify the CM's pin.conf, we must restart the CM to make the changes)
There are many ways to detect memory leaks. In this post we will use Libumem and MDB. I will explain 3 methods (leaving the most practical at last, with a step-by-step example). The libumen library is only available on Solaris 9 update 3 and higher (to verify which system you are using: uname -a).
Method 1
- Run:
export LD_PRELOAD=libumem.so
- Run:
export UMEM_DEBUG=audit
- If possible, put points 1 and 2 in the .profile so you don't have to be executing them every time a new terminal is opened.
- Compile the fm_pol... library with the -g and -lumen flags
- Start CM.
- Execute the opcode to be tested and see the pid (process id) of the CM that is running the opcode.
- In another terminal execute:
echo "::findleaks -d " | mdb -p pid_cm
Method 2
- Run:
export LD_PRELOAD=libumem.so
- Run:
export UMEM_DEBUG=audit
- If possible, put points 1 and 2 in the .profile so you don't have to be executing them every time a new terminal is opened.
- Compile the fm_pol... library with the -g and -lumen flags
- Start CM.
- Execute the opcode to be tested and see the pid (process id) of the CM that is running the opcode.
- In another terminal execute:
gcore pid_cm
The above command will generate a file called core.pid_cm - Run:
mdb core.pid_cm
- Run:
::findleaks
- In case of having memory leaks the output of ::findleaks will show us how many buffers with memory leaks there are.
- From each buffer we can obtain the stack trace by executing ::bufctl_audit for each bufctl address. Note that if they have been more than 16K leaked, the output will include a list of those leaked buffers including the number of bytes and the vmem_seg address. You can obtain stack traces for those buffers by running :: vmem_seg -v for each vmem_seg address.
Method 3
- Run:
export LD_PRELOAD=libumem.so
- Run:
export UMEM_DEBUG=audit
- If possible, put points 1 and 2 in the .profile so you don't have to be executing them every time a new terminal is opened.
- Compile the fm_pol... library with the -g and -lumen flags
- Start CM.
- Create a text file with the input flist of the opcode you want to test.
- Compile testMemory.c with the -g and -lumen flags (at the end of the post is the source code).
- Run the testMemory program with a file containing the input flist and the opcode to be tested. For example: testMemory inFlist.txt PCM_OP_INV_POL_PREP_INVOICE
- The program will connect to BRM, execute the opcode with the flist that was received and generate a core file for the CM that executed the opcode, that's all the program does (run the opcode and generate the core).
- Run:
mdb core.pid_cm
- Run:
::findleaks
- In case of having memory leaks the output of ::findleaks will show us how many buffers with memory leaks there are.
- From each buffer we can obtain the stack trace by executing ::bufctl_audit for each bufctl address. Note that if they have been more than 16K leaked, the output will include a list of those leaked buffers including the number of bytes and the vmem_seg address. You can obtain stack traces for those buffers by running :: vmem_seg -v for each vmem_seg address.
Method 3: Step-by-step example
- The example will be on $PIN_HOME/source/sys/fm_inv_pol : fm_inv_pol.so
- Steps 1, 2 and 3 are already done in my environment (by putting them in the .profile) so I will not do them, but I will show the value of LD_PRELOAD and UMEM_DEBUG.
- Compile with the specified flags: -g and -lumen.
- Start CM.
- As the fm I want to test is invoicng, I am going to make a flist for the PCM_OP_INV_MAKE_INVOICE opcode and I will leave it in the PCM_OP_INV_MAKE_INVOICE.txt file. The flist format is the same we would use for the testnap or Developer Center.
- Compile the testMemory.c file with the -g and -lumen flags
- Run the testMemory program with the file containing the opcode input flist and the name of the opcode: testMemory PCM_OP_INV_MAKE_INVOICE.txt
PCM_OP_INV_MAKE_INVOICE - The program created a connection to the CM, see the memory used by the CM before executing the opcode, execute the opcode with the flist that I passed as a parameter, see the memory used by the CM after using the opcode, generate a core CM file and close the connection to the CM.
- So far we have the core file to analyze with mdb and look for memory leaks. In this example, the file is: core.27895
- Run mdb and then ::findleaks to see how many buffers with memory leaks there are.
- In this case, there are 10 buffers with memory leaks. Now you have to analyze each of those buffers. From each buffer we can obtain the stack tracke by executing ::bufctl_audit for each bufctladdress.
- In the stack of 00635950, we see that our library fm_inv_pol.so leaves memory leaks in the function fm_inv_pol_utils_memory_leak when calling the function pcm_op (last line before the name appears our library)
- So far we have detected what is causing the memory leaks. Now you have to go to the code where the fm_inv_pol_utils_memory_leak function is located, find all calls to PCM_OP (this is where the memory is allocaed and can't be latter released) and see what happens with the fifth parameter of that function (the fifth parameter is the only one that reserves memory in the PCM_OP). If the variable corresponding to the fifth parameter (a **pin_flist_t) is not released (using destroy), putting in another flist (using put), then we have our memory leak.
- For this example, I chose PCM_OP on purpose because I have seen many times that the pin_flist_t (the fifth parameter of the function), where the result of the execution of the opcode is thrown, is not destroyed or a put, or in case of error It is also not being destroyed and that's causing memory leaks.
#include "pcm.h" void PCM_OP( pcm_context_t *pcm_ctxp, int32 opcode, int32 flags, pin_flist_t *in_flistp, pin_flist_t **ret_flistpp, pin_errbuf_t *ebufp);
5th parameter's Online documentation:ret_flistpp
A pointer to a pointer for passing back the return flist. See the individual opcode manual pages for the return flist specifications. All operations produce a return flist with at least the PIN_FLD_POID field on it. Other fields on the return flist depend on the operation being performed. The return flist is passed back even if an error occurred during the operation. It is the responsibility of the caller to destroy the return flist when it is no longer needed.
- So far we have analyzed a buffer of the 10 detected. Let's analyze another one, for example 0040a3b0:
- The call to our fm_inv_pol.so library is not seen here, the memory leak is in the fm_inv.so library in the fm_inv_make_invoice_event_search function when executing pin_flist_create. We don't have the source code from this library, so we can't do anything to solve the memory leak that is causing it.
- We have already analyzed 2 buffers out of 10. The other 8 remain as homework. The mechanics are the same for everyone.
- In the next post (Memory leaks detection - Part II) I will show how from the result of the ::bufctl_audit we can obtain what line of the source code the memory leak is taking place.
pin>echo $LD_PRELOAD libumem.so pin>echo $UMEM_DEBUG audit
pin>make all cc -c -g -xcg92 -lumen -lthread -ldl -lpinsys -shared -DFOR_CM -DFM_INV_POL_DLL -I/export/home/pin/7.5/include -I/export/home/pin/7.5/lib -DPCMCPP_CONST_SAFE -DFOR_CM -DFM_INV_POL_DLL fm_inv_pol_config.c fm_inv_pol_init.c fm_inv_pol_common.c fm_inv_pol_format_view.c fm_inv_pol_prep_invoice.c fm_inv_pol_format_invoice.c fm_inv_pol_format_invoice_doc1.c fm_inv_pol_format_invoice_html.c fm_inv_pol_format_invoice_xml.c fm_inv_pol_format_invoice_xslt.c fm_inv_pol_select.c fm_inv_pol_utils.c fm_inv_pol_post_make_invoice.c fm_inv_pol_config.c: fm_inv_pol_init.c: fm_inv_pol_common.c: fm_inv_pol_format_view.c: fm_inv_pol_prep_invoice.c: fm_inv_pol_format_invoice.c: fm_inv_pol_format_invoice_doc1.c: fm_inv_pol_format_invoice_html.c: fm_inv_pol_format_invoice_xml.c: fm_inv_pol_format_invoice_xslt.c: fm_inv_pol_select.c: fm_inv_pol_utils.c: fm_inv_pol_post_make_invoice.c: ld -o /export/home/pin/7.5/lib/fm_inv_pol.so -G -L/export/home/pin/7.5/lib fm_inv_pol_config.o fm_inv_pol_init.o fm_inv_pol_common.o fm_inv_pol_format_view.o fm_inv_pol_prep_invoice.o fm_inv_pol_format_invoice.o fm_inv_pol_format_invoice_doc1.o fm_inv_pol_format_invoice_html.o fm_inv_pol_format_invoice_xml.o fm_inv_pol_format_invoice_xslt.o fm_inv_pol_select.o fm_inv_pol_utils.o fm_inv_pol_post_make_invoice.o -lm -lpsiu_for_cm
pin>make all cc -c -g -xcg92 -lumem -I/export/home/pin/7.5/include testMemory.c CC -o testMemory -g -xcg92 -lumem -I/export/home/pin/7.5/include -L/export/home/pin/7.5/lib -R:/export/home/pin/7.5/lib: -L/export/home/pin/7.5/lib testMemory.o -lportal -lmta -lpinsys -lsocket -lnsl -lgen -lposix4 -lpthread -ldl
pin>testMemory PCM_OP_INV_MAKE_INVOICE.flist PCM_OP_INV_MAKE_INVOICE Conectado a BRM PID VSZ RSS F S SZ STIME TIME %MEM COMMAND 27895 62872 13896 0 S 7859 19:38:34 00:00 0.1 /export/home/pines/pin14/7.5/bin/cm 27793 62872 36664 0 S 7859 19:38:16 00:01 0.1 /export/home/pines/pin14/7.5/bin/cm CM nuevo, pid = 27895 Opcode PCM_OP_INV_MAKE_INVOICE ejecutado Output Flist: # number of field entries allocated 20, used 1 0 PIN_FLD_POID POID [0] 0.0.0.1 /invoice 2751918 0 PID VSZ RSS F S SZ STIME TIME %MEM COMMAND 27895 91544 43320 0 S 11443 19:38:34 00:02 0.2 /export/home/pines/pin14/7.5/bin/cm 27793 62872 36664 0 S 7859 19:38:16 00:01 0.1 /export/home/pines/pin14/7.5/bin/cm Ejecutando: gcore 27895 gcore: core.27895 dumped
pin>mdb core.27895 Loading modules: [ libumem.so.1 libc.so.1 libuutil.so.1 ld.so.1 ] > ::findleaks mdb: [fc000000, fc564000): couldn't read 16384 bytes at fc000000: no mapping for address CACHE LEAKED BUFCTL CALLER 000a8008 1 0040a2e0 libcmpin.so`pcmmem_malloc_flistflds+0x34 000a8008 1 0058da18 libcmpin.so`pcmmem_malloc_flistflds+0x34 000a8008 1 0040a3b0 libcmpin.so`pcmmem_malloc_flistflds+0x34 000a8008 1 005e6220 libcmpin.so`pcmmem_malloc_flistflds+0x34 000a4008 1 0040ab68 libcmpin.so`pcmmem_malloc_flisthdr+0x34 000a4008 1 004001a0 libcmpin.so`pcmmem_malloc_flisthdr+0x34 000a4008 1 005814c8 libcmpin.so`pcmmem_malloc_flisthdr+0x34 000a4008 1 005e6970 libcmpin.so`pcmmem_malloc_flisthdr+0x34 0006c008 1 00635950 libcmpin.so`pcmmem_malloc_poid+0x34 0006c008 1 0064dd40 libcmpin.so`pcmmem_malloc_poid+0x34 ---------------------------------------------------------------------- Total 10 buffers, 2880 bytes
> 00635950::bufctl_audit
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
635950 6319e0 9ebf699f553207 1
6c008 0 0
libumem.so.1`umem_cache_alloc+0x210
libumem.so.1`umem_alloc+0x60
libumem.so.1`malloc+0x28
libcmpin.so`pcmmem_malloc_poid+0x34
libcmpin.so`pcpxdr_poid+0xd0
libcmpin.so`pcpxdr_fld_list+0xe04
libcmpin.so`pcpxdr_op_decode+0x8c
libcmpin.so`pcp_receive_no_trans_cleanup+0x128
libcmpin.so`pcm_op_ex+0x9fc
libcmpin.so`pcm_op+0x1c
fm_inv_pol.so`fm_inv_pol_utils_memory_leak+0x3b0
fm_inv_pol.so`fm_inv_pol_utils_add_fields+0xf0
fm_inv_pol.so`fm_inv_pol_prep_invoice+0x124
fm_inv_pol.so`op_inv_pol_prep_invoice+0xf4
libcmpin.so`cm_pre_pcm_op+0x14f4
> 0040a3b0::bufctl_audit
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
40a3b0 4138c0 9ebf65e1425464 1
a8008 0 0
libumem.so.1`umem_cache_alloc+0x210
libumem.so.1`umem_alloc+0x60
libumem.so.1`malloc+0x28
libcmpin.so`pcmmem_malloc_flistflds+0x34
libcmpin.so`pini_flist_grow+0x48
libcmpin.so`pin_flist_create_with_size+0xc4
libcmpin.so`pin_flist_create+0x48
fm_bill_utils.so`fm_bill_utils_search_exchange_rate_from_db+0x420
fm_bill.so`fm_bill_init_config_currency_conversionrates+0x6c
fm_bill.so`fm_bill_init+0x1ec
libcm_main.so`mainThread+0x323c
main+8
_start+0x108
#include <stdio.h>
#include <stdio.h>
#include <string.h>
#include "pcm.h"
#include "pinlog.h"
#include <unistd.h>
#include <dlfcn.h>
#define MAX_PROCESS 256
#define STRING_LENGH 256
//ejecuta el comando recibido por parametro
void
ejecutar_comando(
char *comando);
//crea un flist que se encuentra contenido en un archivo
pin_flist_t*
create_flist_from_file(
char *file_name,
int64 db,
pin_errbuf_t *ebufp,
int print);
//Obtiene los CMs que se encuentran en ejecucion actualmente
void get_CMs_actuales(
char array_CMs[MAX_PROCESS][STRING_LENGH]);
//Obtiene los CMs que se encuentran en ejecucion actualmente, pero que no estaban en ejecucion antes
void get_CMs_nuevos(
char array_CMs_anteriores[MAX_PROCESS][STRING_LENGH],
char array_CMs[MAX_PROCESS][STRING_LENGH]);
//Genero el archivo core para cada uno de los CMs recibidos
void generar_cores(
char array_CMs[MAX_PROCESS][STRING_LENGH]);
int main(
int argc,
char *argv[])
{
pcm_context_t *ctxp;
pin_flist_t *out_flistp;
pin_flist_t *in_flistp;
pin_opcode_t opcode;
pin_errbuf_t ebuf;
int64 db = 1;
char *file_name = NULL;
char *opname = NULL;
char array_CMs_anteriores[MAX_PROCESS][STRING_LENGH];
char array_CMs_nuevos[MAX_PROCESS][STRING_LENGH];
if(argc != 3){
fprintf(stderr, "\nModo de uso: archivo_flist opcode_name\n\n");
exit(1);
}
file_name = argv[1];
opname = argv[2];
opcode = pcm_opname_to_opcode(opname);
if(opcode == 0){
fprintf(stderr, "\nError, el opcode %s no existe\n\n", opname);
exit(1);
}
PIN_ERR_CLEAR_ERR(&ebuf);
//leo el flist que se encuentra en el archivo
in_flistp = create_flist_from_file(file_name, db, &ebuf, 0);
if (PIN_ERR_IS_ERR(&ebuf)) {
fprintf(stderr, "Error al armar el flist del archivo %s\n", file_name);
PIN_FLIST_DESTROY_EX(&in_flistp, NULL);
exit(1);
}
if(in_flistp == NULL){
fprintf(stderr, "Error al armar el flist del archivo %s\n", file_name);
exit(1);
}
//Obtengo los CMs que se encuentran en ejecucion actualmente
get_CMs_actuales(array_CMs_anteriores);
//Me conecto a BRM, lo que generara un nuevo CM
PCM_CONNECT(&ctxp, &db, &ebuf);
if (PIN_ERR_IS_ERR(&ebuf)) {
fprintf(stderr, "Error al ejecutar PCM_CONNECT\n");
PIN_FLIST_DESTROY_EX(&in_flistp, NULL);
exit(1);
}
fprintf(stdout, "\n\nConectado a BRM\n\n");
//Muestro el estado de los CMs antes de ejecutar el opcode
ejecutar_comando("ps -eo pid,vsz,rss,f,s,osz,stime,time,pmem,comm | head -1");
ejecutar_comando("ps -eo pid,vsz,rss,f,s,osz,stime,time,pmem,comm | grep $PIN_HOME/bin/cm");
//Obtengo los CMs que se iniciarion despues que la ultima vez que los revise, aqui estara el que ejecutara el opcode
get_CMs_nuevos(array_CMs_anteriores, array_CMs_nuevos);
//Ejecuto el opcode con el flist del archivo
PCM_OP(ctxp, opcode, 0, in_flistp, &out_flistp, &ebuf);
PIN_FLIST_DESTROY_EX(&in_flistp, NULL);
if (PIN_ERR_IS_ERR(&ebuf)) {
fprintf(stderr, "\n\nError ejecutando el opcode %s\n", opname);
PIN_ERR_LOG_EBUF(PIN_ERR_LEVEL_ERROR, "main error", &ebuf);
PIN_ERR_CLEAR_ERR(&ebuf);
} else {
fprintf(stdout, "\n\nOpcode %s ejecutado \n", opname);
fprintf(stdout, "\nOutput Flist:\n");
PIN_FLIST_PRINT(out_flistp, NULL, &ebuf);
}
PIN_FLIST_DESTROY_EX(&out_flistp, NULL);
fprintf(stdout, "\n\n");
//Muestro el estado de los CMs despues de ejecutar el opcode
ejecutar_comando("ps -eo pid,vsz,rss,f,s,osz,stime,time,pmem,comm | head -1");
ejecutar_comando("ps -eo pid,vsz,rss,f,s,osz,stime,time,pmem,comm | grep $PIN_HOME/bin/cm");
//genero los archivos core para todos los CMs nuevos, entre los cuales se encuentra el que ejecuto este opcode
generar_cores(array_CMs_nuevos);
//cierro el contex
PCM_CONTEXT_CLOSE(ctxp, 0, &ebuf);
if (PIN_ERR_IS_ERR(&ebuf)) {
fprintf(stderr, "Error ejecutando PCM_CONTEXT_CLOSE\n");
exit(1);
}
return 0;
}
//crea un flist que se encuentra contenido en un archivo
pin_flist_t*
create_flist_from_file(
char *file_name,
int64 db,
pin_errbuf_t *ebufp,
int print)
{
char *buffer;
char linea[1048];
pin_flist_t *in_flistp = NULL;
long length = 0;
FILE *file = fopen(file_name, "r");
if (file) {
fseek(file, 0, SEEK_END);
length = ftell(file) + 1;
if(length > 15){
fseek(file, 0, SEEK_SET);
buffer = (char*)calloc(length, sizeof(char));
if (buffer != NULL) {
while (feof(file) == 0) {
fgets(linea, sizeof(linea), file);
if(strlen(linea) > 15 && feof(file) == 0 ){
//solo me quedo con las lineas de mas de 15 caracteres
strcat(buffer, linea);
}
}
} else {
fprintf(stderr, "\nError, calloc de buffer devolvio NULL \n");
}
} else {
//el archivo debe tener por lo menos los datos de un poid, si es demasiado chico ni eso tiene
fprintf(stderr, "\nError, el contenido del archivo no es valido\n");
}
fclose(file);
} else {
fprintf(stderr, "\nError al abrir el archivo %s\n", file_name);
return NULL;
}
if (buffer != NULL) {
PIN_STR_TO_FLIST(buffer, db, &in_flistp, ebufp);
if(print != 0){
fprintf(stdout, "\nFlist leido del archivo %s\n", file_name);
PIN_FLIST_PRINT(in_flistp, NULL, ebufp);
}
free(buffer);
}
return in_flistp;
}
//ejecuta el comando recibido por parametro
void
ejecutar_comando(char *comando)
{
char buffer[256];
char *result = NULL;
FILE *pipe = popen(comando, "r");
if (!pipe) {
fprintf(stderr, "\nError al ejecutar el comando: %s\n", comando);
return;
}
result = (char*)malloc(256*sizeof(buffer));
if(result == NULL){
fprintf(stderr, "\nError, malloc de result devolvio NULL \n");
return;
}
memset(result, '\0', sizeof(result));
while(!feof(pipe)) {
if(fgets(buffer, sizeof(buffer)-1, pipe) != NULL){
strcat(result, buffer);
}
}
pclose(pipe);
fprintf(stdout, "%s", result);
free(result);
}
//Obtiene los CMs que se encuentran en ejecucion actualmente
void get_CMs_actuales(
char array_CMs[MAX_PROCESS][STRING_LENGH])
{
char *comando = "ps -eo pid,vsz,rss,f,s,osz,stime,time,pmem,comm | grep $PIN_HOME/bin/cm";
FILE *pipe = popen(comando, "r");
int i = 0;
if (!pipe) {
fprintf(stderr, "\nError al ejecutar el comando: %s\n", comando);
return;
}
while(!feof(pipe)) {
if(fgets(array_CMs[i], sizeof(array_CMs[i])-1, pipe) != NULL){
i++;
}
if (i >= MAX_PROCESS) {
fprintf(stdout, "\n\nSe supero la cantidad de procesos definidos, no se mostraran todos los procesos, solamente %d procesos\n\n", MAX_PROCESS);
break;
}
}
pclose(pipe);
}
//Obtiene los CMs que se encuentran en ejecucion actualmente, pero que no estaban en ejecucion antes
void get_CMs_nuevos(
char array_CMs_anteriores[MAX_PROCESS][STRING_LENGH],
char array_CMs[MAX_PROCESS][STRING_LENGH])
{
char *comando = "ps -eo pid,vsz,rss,f,s,osz,stime,time,pmem,comm | grep $PIN_HOME/bin/cm";
FILE *pipe = popen(comando, "r");
char aux[STRING_LENGH];
char *sub_str = NULL;
char *str = NULL;
char *str_ps = NULL;
int i = 0;
if (!pipe) {
fprintf(stderr, "\nError al ejecutar el comando: %s\n", comando);
return;
}
fprintf(stdout, "\n");
while(!feof(pipe)) {
if(fgets(aux, sizeof(aux)-1, pipe) != NULL){
str_ps = strtok_r(aux, " ", &sub_str);
if(str_ps != NULL){
int encontrado = 0;
for(int j = 0; (j < MAX_PROCESS) && (array_CMs_anteriores[j] != NULL) && (strlen(array_CMs_anteriores[j]) > 0) && (encontrado == 0); j++){
str = strtok_r(array_CMs_anteriores[j], " ", &sub_str);
if(strcmp(str_ps, str) == 0){
//es un CM de los anteriories al context open
encontrado = 1;
}
}
if(encontrado == 0){
fprintf(stdout, "CM nuevo, pid = %s\n", str_ps);
strcpy(array_CMs[i], aux);
i++;
}
}
}
if (i >= MAX_PROCESS) {
fprintf(stdout, "\n\nSe supero la cantidad de procesos definidos, no se mostraran todos los procesos, solamente %d procesos\n\n", MAX_PROCESS);
break;
}
}
pclose(pipe);
}
//Genero el archivo core para cada uno de los CMs recibidos
void
generar_cores(
char array_CMs[MAX_PROCESS][STRING_LENGH])
{
char *sub_str = NULL;
char *cm_pid = NULL;
char comando[20];
for(int i = 0; (i < MAX_PROCESS) && (array_CMs[i] != NULL) && (strlen(array_CMs[i]) > 0); i++){
cm_pid = strtok_r(array_CMs[i], " ", &sub_str);
//generando el archivo core para el cm_pid
sprintf(comando, "gcore %s", cm_pid);
fprintf(stdout, "\n\nEjecutando: %s\n\n", comando);
ejecutar_comando(comando);
}
}
Martín Falconi
No hay comentarios.:
Publicar un comentario