Vlad Bogolin
AI & ML interests
Articles
Organizations
vladbogo's activity
Key points:
1️⃣ It uses pre-trained diffusion models to enable precise and high-fidelity object swapping in images.
2️⃣Targeted variable swapping ensures perfect background preservation while swapping specific areas.
3️⃣SwapAnything achieves good results in single-object, multi-object, partial-object, and cross-domain swapping tasks.
Paper: SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing (2404.05717)
Project page: https://swap-anything.github.io
Congrats to the authors for their work!
Key Points:
* Prompts LLMs with hundreds of examples of harmful behavior formatted as a dialogue
* Generates malicious examples using an uninhibited "helpful-only" model
* Effective at jailbreaking models like Claude 2.0, GPT-3.5, GPT-4
* Standard alignment techniques provide limited protection against long context attacks
Paper: https://www.anthropic.com/research/many-shot-jailbreaking
More details in my blog: https://huggingface.co/blog/vladbogo/many-shot-jailbreaking
Congrats to the authors for their work!
Keypoints:
* Uses an LLM to generate diverse synthetic queries and tasks from web passages
* Refines the data by retrieving candidate passages and relabeling positives/negatives using the same LLM
* Achieves very good results on the Massive Text Embedding Benchmark, where compact 256D Gecko outperforms 768D models.
* 768D Gecko achieves state-of-the-art performance competing with models a lot larger larger.
Paper: Gecko: Versatile Text Embeddings Distilled from Large Language Models (2403.20327)
More details in my blog: https://huggingface.co/blog/vladbogo/gecko
Congrats to the authors for their work!
Keypoints:
* SAFE (Search-Augmented Factuality Evaluator) is an automated method using an LLM agent to evaluate factuality
* It also introduces LongFact, a 2,280 prompt set spanning 38 topics to test open-domain factual knowledge
* SAFE achieves a 72% humans agreement while being 20x cheaper. It also wins 76% of the disagreements measured on a small scale experiment where a more thorough human procedure (researchers + full internet search) was used.
* Larger models like GPT-4, Claude Opus and Gemini Ultra tend to exhibit better long-form factuality.
Paper: Long-form factuality in large language models (2403.18802)
Code and data: https://github.com/google-deepmind/long-form-factuality
Congrats to the authors for their work!
Keypoints:
* Introduces the 373k Visual CoT dataset with bounding box annotations highlighting essential image regions
* Proposes a multi-turn pipeline for focusing on relevant visual inputs
* Achieves strong results on multi-modal benchmarks
Paper: Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models (2403.16999)
Code, data and other resources: https://github.com/deepcs233/Visual-CoT
Congrats to the authors for their work!
Blog: https://x.ai/blog/grok-os
Code: https://github.com/xai-org/grok
Model: xai-org/grok-1
Weights: magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce
Key points:
* Enables precise animation of selected image regions with just a user click and a concise motion description.
* Achieves promising results for generating localized animations.
Paper: Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts (2403.08268)
Congrats to the authors for their work!
Key Points:
* Overcomes data limitations by generating high-quality synthetic image-caption pairs, reducing reliance on costly human annotations.
* Achieves competitive results on image captioning tasks using 40x less paired data than state-of-the-art methods.
Paper: Synth$^2$: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings (2403.07750)
Congrats to the authors for their work!
Key points:
* Discovers significant redundancy across layers in LLMs, with some layers playing a negligible role for the final performance.
* Defines a new metric called Block Influence (BI) to quantify the importance of each layer in an LLM.
* Removes layers with low BI scores, achieving up to 25% reduction in parameters and computation while maintaining 92% of the LLM's performance.
Congrats to the authors for their work!
Paper: ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2403.03853)
Key points:
* Defines multiple metrics to assess image-text quality from different perspectives like object details, text quality, and semantic understanding.
* Leverages GPT-4 and GPT-4V to construct high-quality instruction data for fine-tuning open-source MLMs as effective data filters.
* Fine-tuned MLM filters generate more precise scores, leading to better filtered data and improved performance of pre-trained models on various downstream tasks.
Congrats to the authors for their work!
Paper: Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters (2403.02677)
Code: https://github.com/Victorwz/MLM_Filter
Dataset: weizhiwang/mlm_filter_instructions
Model: weizhiwang/mlm-filter-llava-13b-gpt4v
Key Points:
* Proposes two methods - LoRA Switch and LoRA Composite - that activate/combine LoRAs during the denoising process rather than merging weights
* LoRA Switch cycles through different LoRAs at each step, while LoRA Composite averages guidance from all LoRAs simultaneously
Paper: Multi-LoRA Composition for Image Generation (2402.16843)
Project page: https://maszhongming.github.io/Multi-LoRA-Composition
Congrats to the authors for their work!
Key Points:
* Introduces the Design2Code task and benchmark for converting webpage screenshots into code, aiming to automate front-end web development.
* Evaluates multimodal LLMs using comprehensive metrics for visual similarity and element matching.
* GPT-4V outperforms other models in terms of visual resemblance and content accuracy, with generated webpages often preferred over the original references.
Paper: Design2Code: How Far Are We From Automating Front-End Engineering? (2403.03163)
Project page: https://salt-nlp.github.io/Design2Code/
Dataset: SALT-NLP/Design2Code
Congrats to the authors for their work!
Keypoints:
* Outperforms state-of-the-art vision transformers like DiT, SiT, DeiT3, and Swin on multiple benchmarks and tasks.
* Leverages Auto-Scaled 2D Rotary Positional Embeddings (AS2DRoPE) to handle variable input resolutions efficiently.
* Serves as a powerful, unified modeling framework for vision generation and understanding tasks.
Paper: VisionLLaMA: A Unified LLaMA Interface for Vision Tasks (2403.00522)
GitHub repo: https://github.com/Meituan-AutoML/VisionLLaMA
Congrats to the authors for their work!
Key Points:
* Automatic Caption Generation: Utilizes an automatic pipeline with multiple cross-modality teacher models to generate captions for video clips.
* Fine-tuned Caption Selection: Employs a fine-tuned retrieval model to select the most appropriate caption from multiple candidates for each video clip.
* Improved Performance: Pre-training on Panda-70M shows significant performance gains in video captioning, text-video retrieval, and text-driven video generation.
Paper: Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2402.19479)
Project page: https://snap-research.github.io/Panda-70M/
Code: https://github.com/snap-research/Panda-70M
Congrats to the authors @tschen , @aliaksandr-siarohin et al. for their work!
Keypoints:
* Dataset: It introduces "ConflictingQA," a dataset of controversial questions and real-world evidence paragraphs supporting both "yes" and "no" answers.
* Convincingness Metric: It uses the "paragraph win rate" - when shown two conflicting paragraphs, this measures how often a model predicts the answer that aligns with a given paragraph's stance.
* Current models rely on the relevance of the content to the query, while largely ignoring stylistic features such as whether a text contains scientific references or if it is written with a neutral tone.
Congrats to the authors for their work!
Paper: What Evidence Do Language Models Find Convincing? (2402.11782)
Code: https://github.com/AlexWan0/rag-convincingness
totally agree. Can't wait to see what comes next
Keypoints:
* Genie leverages a spatiotemporal video tokenizer, an autoregressive dynamics model, and a latent action model to generate controllable video environments.
* The model is trained on video data alone, without requiring action labels, using unsupervised learning to infer latent actions between frames.
* The method restricts the size of the action vocabulary to 8 to ensure that the number of possible latent actions remains small.
* The dataset used for training is generated by filtering publicly available internet videos with specific criteria related to 2D platformer games for a total of 6.8M videos used for training.
Paper: Genie: Generative Interactive Environments (2402.15391)
Project page: https://sites.google.com/view/genie-2024/
More detailed overview in my blog: https://huggingface.co/blog/vladbogo/genie-generative-interactive-environments
Congrats to the authors for their work!
Agree! I don't think it's at all feasible to handle these types of problems/attacks at a provider level. So, as you said, I think that new open-source defensive tool chains will emerge. However, I think the paper makes a good step towards showcasing some of the current capabilities and can enable further research both for finding more complex attacks and also mitigations.
The key findings are:
* LoRA fine-tuning (LFT) preserves the pre-training token distribution while SFT doesn't. This indicates that using LFT, post fine-tuning the model still heavily relies on the pre-training and doesn't acquire new information.
* Dataset scaling is ineffective for LFT - experiments show that scaling the dataset size 52x or even 326x doesn't improve the performance.
* LoRA fine-tuning mainly enhances response initiation and style without substantial knowledge enhancement.
* Full-parameter fine-tuning tends to degrade LLM knowledge base and increase hallucination occurrences.
* Popular other methods and adjustments fail to significantly outperform simple LoRA fine-tuned models in terms of conversational quality and accuracy.
Congrats to the authors @Sreyan88 and others for their work!
Paper: A Closer Look at the Limitations of Instruction Tuning (2402.05119)
Key points:
* It uses a LLM integrated with Playwright, a headless web browser, enabling automated web interactions through function calling.
* It gives access to the LLM to 7 web hacking documents and planning capabilities through specific prompting, without disclosing the exact methods to prevent misuse.
GPT-4 achieves a 73.3% success rate on the tested vulnerabilities, emphasizing the potential cybersecurity risks posed by advanced LLMs. Other open models cannot yet perform these types of attacks (results in screenshot).
Congrats to the authors for their work!
Paper: LLM Agents can Autonomously Hack Websites (2402.06664)
Key points:
* It employs a two-stage training approach, initially aligning video and text encoders, followed by an enhanced video-only masked autoencoding process to learn appearance and motion.
* It achieves superior performance in a wide array of tasks, such as general video understanding, zero-shot video-text retrieval, video captioning, QA, and computer vision for science, having top performance on 30 out of 33 benchmarks.
Congrats to the authors for their work!
Paper: VideoPrism: A Foundational Visual Encoder for Video Understanding (2402.13217)
A more detailed overview can be found in my blog: https://huggingface.co/blog/vladbogo/rephrasing-the-web. Feedback is appreciated!
Thanks! Appreciate it!
Key aspects:
* Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
* Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.
Congrats to the authors for their work!
Paper: Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2401.16380)
For anyone interested in more information, I also written a blog that highlights more aspects: https://huggingface.co/blog/vladbogo/reformatted-alignment. Feedback is appreciated!
Key points:
* REALIGN has three steps: criteria definition, retrieval augmentation, and response reformatting
* It rewrites pairs (query, response) to enhance data quality for fine-tuning LLMs.
* It has shown significant improvements in general alignment, math reasoning and other tasks.
Congrats to the authors for their work!
Paper: Reformatted Alignment (2402.12219)
Code: https://github.com/GAIR-NLP/ReAlign
As far as I could find it’s not yet available. Hopefully the authors will release it soon 🤞
Key aspects of the paper:
• It introduces Spectral DeTuning for reversing Low-rank Adaptation (LoRA) fine-tuning, targeting original weight restoration through spectral analysis.
• LoWRA Bench Dataset: It introduces a dataset for testing Spectral DeTuning across various models and tasks, featuring extensive layers for comprehensive evaluation.
• It reveals LoRA fine-tuned models' vulnerability to weight recovery attacks, questioning the security of fine-tuning modifications.
Congrats to the authors for their work!
Paper: Recovering the Pre-Fine-Tuning Weights of Generative Models (2402.10208)
Dataset: Eliahu/LoWRA-Bench
Project page: https://vision.huji.ac.il/spectral_detuning/
Code: https://github.com/eliahuhorwitz/Spectral-DeTuning
Thanks for trying it out! I looked on the logs and it seems that it hangs fetching web search results. We've been getting these types of errors from time to time, so I temporary disabled web search until I can find a better fix. Now it should be faster (still should take around 1 minute), so hopefully you'll get some results. This means that now it relies on Wikipedia only to find evidence for the identified claims.
There were also some gpt-4 processing errors, so I would kindly ask you if you still encounter errors to please send me the paragraph that you used so I can further debug. If you prefer, you can also email it to me at [email protected].
Thanks again!
Feedback is appreciated!
The research investigates how the order of premises affects LLMs in logical and mathematical reasoning tasks, challenging the assumption that premise sequence is irrelevant to the outcome.
Key Findings:
* Logical Reasoning: LLMs perform best when premises are in a forward order that aligns with the proof's progression. Deviations from this order result in significant performance drops.
* Mathematical Reasoning: The introduction of the R-GSM benchmark shows a similar sensitivity in LLMs.
Congrats to the authors for their work!
Paper: Premise Order Matters in Reasoning with Large Language Models (2402.08939).
Key aspects of Lumos include:
* Hybrid Computing: Utilizes a combination of on-device and cloud computing to process inputs, aiming to reduce latency.
* STR Components:
* Region of Interest (ROI) Detection: Focuses on text-rich areas within images for optimized text extraction.
* Text Detection and Recognition: Ensures high-quality text recognition within the ROI.
* Reading Order Reconstruction: Arranges recognized text to mimic natural reading order, essential for context understanding.
Lumos demonstrates significant improvement with 80% accuracy in question-answering benchmarks and a low word error rate.
Paper: Lumos : Empowering Multimodal LLMs with Scene Text Recognition (2402.08017)
Congrats to the authors for their work!
The system has three components:
* Planner: It takes complex user requests and breaks them down into manageable subtasks for efficient execution.
* Configurator: It prepares tasks for execution based on your preferences and available commands using a memory mechanism.
* Actor: It executes the tasks and learns from feedback, ensuring continuous improvement.
FRIDAY outperforms other methods on GAIA, a comprehensive benchmark. To answer the questions from GAIA, the agents need skills to calculate numbers, browse the web, process video and speech signal and others.
Resources:
* Paper: OS-Copilot: Towards Generalist Computer Agents with Self-Improvement (2402.07456)
* Project GitHub: https://github.com/OS-Copilot/FRIDAY
* Project page: https://os-copilot.github.io/
Congrats to the authors Wu, Zhiyong et al. for their work!